Under-display camera and sensor control

ABSTRACT

An under-display camera is positioned underneath a display of a mobile device. The under-display camera captures an image using light passing through a portion of the display. The mobile device displays a display image on the display, the display image based on the image. The mobile device displays an indicator overlaid over the display image on an indicator area of the display that overlaps with the portion of the display. The indicator may identify the position of the camera. The mobile device can compensate for occlusion of the camera by continuing to display a previous display image if a more recently captured image includes an occlusion. The mobile device can give users alternate ways to select areas of the display image to avoid camera occlusion, for instance using hardware buttons and/or touchscreen interface elements.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional ApplicationNo. 17/068,412, filed, Oct. 12, 2020, which is hereby incorporated byreference, in its entirety and for all purposes.

FIELD

This application is related to image capture and image processing. Morespecifically, this application relates to technologies and techniques toreceive user inputs to control an under-display camera and/orunder-display sensor and compensate for occlusions of the under-displaycamera and/or under-display sensor.

BACKGROUND

Some mobile devices include a front-facing camera that faces a userwhile the user holds the mobile device. The front-facing camera can beadjacent to a display screen of a mobile device. This allows the mobiledevice to use the front-facing camera and the adjacent display screenfor videoconferencing, for example. Because mobile devices are generallysmall for portability, front-facing cameras are often positioned withina bezel around a display screen. Some mobile device include displayscreens with notches or cutouts within which the display cannot displayanything, and within which the front-facing camera is positioned. Userscan generally see where the front-facing camera is positioned when thefront-facing camera is positioned in a bezel, notch, or cutout. However,bezels, notches, and cutouts take up valuable real estate on the frontsurface of a mobile device, and prevent a mobile device from having adisplay screen that occupies the full front surface of the mobiledevice.

SUMMARY

Systems and techniques are described herein for using an under-displaycamera and/or under-display sensor positioned underneath a display of adevice (e.g., a mobile device or other device). The device can displayan indicator indicating the position of the under-display camera and/orunder-display sensor on the display. The device can compensate forocclusions of the under-display camera and/or under-display sensor. Forexample, such occlusions may be caused by a user or input devicetouching the display in a way that occludes the under-display cameraand/or under-display sensor, a user or input device hovering over thedisplay in a way that occludes the under-display camera and/orunder-display sensor, or a user or input device otherwise being closeenough to the display to occlude the under-display camera and/orunder-display sensor. The device can control the under-display cameraand/or under-display sensor based on the inputs. The device can includeuser interface (UI) elements that allow for selection of an area of animage without touching, hovering over, or otherwise covering the area ofthe image, for instance where such an input at the area of the imagewould occlude the under-display camera and/or under-display sensor.

In one example, an apparatus for image processing is provided. Theapparatus includes a memory. The apparatus includes one or moreprocessors coupled to the memory and configured to perform a method. Themethod includes receiving an image captured by a camera based onincident light received by the camera. The camera is positioned relativeto a display to receive the incident light that passes through a portionof the display before reaching the camera. The method includesdisplaying, on the display, a display image that is based on the image.The method includes displaying an indicator overlaid over the displayimage at an indicator area of the display while the display isdisplaying the display image. The indicator area includes at least asubset of the portion of the display.

In another example, an apparatus for image processing is provided. Theapparatus includes one or more memory units storing instructions. Theapparatus includes one or more processors that execute the instructions,wherein execution of the instructions by the one or more processorscauses the one or more processors to perform a method. The methodincludes receiving an image captured by a camera based on incident lightreceived by the camera. The camera is positioned relative to a displayto receive the incident light that passes through a portion of thedisplay before reaching the camera. The method includes displaying, onthe display, a display image that is based on the image. The methodincludes displaying an indicator overlaid over the display image at anindicator area of the display while the display is displaying thedisplay image. The indicator area includes at least a subset of theportion of the display.

In another example, a method of image processing is provided. The methodincludes receiving image data captured by an image sensor. The methodincludes receiving an image captured by a camera based on incident lightreceived by the camera. The camera is positioned relative to a displayto receive the incident light that passes through a portion of thedisplay before reaching the camera. The method includes displaying, onthe display, a display image that is based on the image. The methodincludes displaying an indicator overlaid over the display image at anindicator area of the display while the display is displaying thedisplay image. The indicator area includes at least a subset of theportion of the display.

In another example, an non-transitory computer readable storage mediumhaving embodied thereon a program is provided. The program is executableby a processor to perform a method of image processing. The methodincludes receiving an image captured by a camera based on incident lightreceived by the camera. The camera is positioned relative to a displayto receive the incident light that passes through a portion of thedisplay before reaching the camera. The method includes displaying, onthe display, a display image that is based on the image. The methodincludes displaying an indicator overlaid over the display image at anindicator area of the display while the display is displaying thedisplay image. The indicator area includes at least a subset of theportion of the display.

In another example, an apparatus for image processing is provided. Theapparatus includes a memory. The apparatus includes means for receivingan image captured by a camera based on incident light received by thecamera. The camera is positioned relative to a display to receive theincident light that passes through a portion of the display beforereaching the camera. The apparatus includes means for displaying, on thedisplay, a display image that is based on the image. The apparatusincludes means for displaying an indicator overlaid over the displayimage at an indicator area of the display while the display isdisplaying the display image. The indicator area includes at least asubset of the portion of the display.

In another example, a method of image processing is provided. The methodincludes receiving an image captured by a camera. The camera ispositioned underneath an area of a display and captures the image usinglight passing through the display. The method includes displaying theimage on the display. The method includes setting an image processingproperty to a particular setting that is determined based on the area ofthe display. The image processing property is one of a white balance, ablack level compensation, a gamma correction, a gain, a colorcorrection, a color saturation, a noise filter, a contrast control, abrightness control, a tone mapping, a sharpness, a blurring, or a redeye correction.

In another example, a method of image processing is provided. The methodincludes receiving an image captured by a camera. The camera capturesthe image based on incident light received by the camera. The camera ispositioned relative to a touchscreen display to receive the incidentlight that passes through a portion of the touchscreen display beforereaching the camera. The method includes displaying, on the touchscreen,a display image based on the image. The method includes displaying anarea selection user interface (UI) element on the touchscreen. Themethod includes receiving a touch input to the touchscreen display at anarea selection input area that overlaps with the area selection UIelement. The method includes selecting a selection area of the displayimage in response to receiving the touch input. The method includessetting a camera property of the camera to a setting that is determinedbased on the selection area of the display image in response toselection of the selection area, wherein the camera property of thecamera includes at least one of an image capture property or an imageprocessing property.

In some aspects, the indicator area of the display is at least as largeas a lens of the camera. In some aspects, the indicator is at least aslarge as a lens of the camera.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: receiving an input, wherein theindicator is displayed overlaid over the indicator area in response toreceipt of the input. In some aspects, the input includes at least oneof a touch of a touch-sensitive touchscreen surface of the display, apredefined gesture across the touch-sensitive touchscreen surface of thedisplay, a hover over the display, a predefined gesture over thedisplay, a voice input reciting a predefined phrase recorded by amicrophone.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: generating the display image bymodifying a size of the image using at least one of scaling andcropping. In some aspects, modifying the size of the image includesmodifying the size of the image such that the display image is displayedon the display other than on the indicator area.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: generating the display image byprocessing the image, wherein processing the image includes modifying atleast one of a brightness, a contrast, a color saturation, a tone, awhite balance, a black balance, a level, a gamma correction, a gain, acolor correction, a noise filter, a sharpness, a blurring, and a red eyecorrection.

In some aspects, wherein the camera is positioned adjacent to thedisplay along an axis that is perpendicular to a planar surface of thedisplay. In some aspects, the indicator includes a shape around at leastpart of the portion of the display. In some aspects, displaying theindicator includes causing a plurality of pixels in an indicator area todisplay a color. In some aspects, the color is black. In some aspects,the indicator is animated.

In some aspects, displaying the indicator overlaid over the displayimage at the indicator area while the display is displaying the displayimage includes modifying the display image to merge the indicator withthe display image. In some aspects, the display is a touchscreen.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: displaying an image capture userinterface (UI) element over an image capture UI area on the display;receiving an input at the image capture UI area, wherein the input isone of a touch input, a hover input, and a gesture input; receiving asecond image captured by the camera in response to receiving the inputat the image capture UI area; and storing the second image to anon-transitory computer-readable storage medium.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: receiving an input at an areaselection user interface (UI) area, wherein the input is one of a touchinput, a hover input, and a gesture input; and selecting a selectionarea of the display image in response to receiving the input at the areaselection UI area, wherein the area selection UI area is distinct fromthe selection area of the display image. In some aspects, the selectionarea of the display image includes at least a portion of the indicatorarea. In some aspects, the methods, apparatuses, and computer-readablemedium described above further comprise: displaying an area selection UIelement at the area selection UI area. In some aspects, the methods,apparatuses, and computer-readable medium described above furthercomprise: generating the area selection UI element to depict a copy ofat least a portion of the selection area.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: setting an image capture parameter toa particular setting that is determined based on the selection area ofthe display image in response to selection of the selection area,wherein the image capture parameter includes at least one of a focus andan exposure parameter. In some aspects, the methods, apparatuses, andcomputer-readable medium described above further comprise: setting animage processing property to a particular setting that is determinedbased on the selection area of the display image in response toselection of the selection area, wherein the image processing propertyis one of a white balance, a black level compensation, a gammacorrection, a gain, a color correction, a color saturation, a noisefilter, a contrast control, a brightness control, a tone mapping, asharpness, a blurring, and a red eye correction.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: receiving a second image captured bythe camera after capture of the image; determining that the second imageis occluded by an occlusion; and displaying the display image on thedisplay in response to determining that the second image is occluded bythe occlusion. In some aspects, the methods, apparatuses, andcomputer-readable medium described above further comprise: receiving aninput, wherein the occlusion is associated with receipt of the input,wherein the input is one of a touch input, a hover input, and a gestureinput; selecting a selection area of the display image based on theinput; determining a setting based on the selection area of the displayimage; applying the setting; and receiving a third image captured by thecamera while the setting is applied and after capture of the secondimage.

In some aspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: receiving a second image captured by asecond camera based on secondary incident light received by the secondcamera, wherein the second camera is positioned relative to the displayto receive the secondary incident light that passes through a secondportion of the display before reaching the second camera. In someaspects, the methods, apparatuses, and computer-readable mediumdescribed above further comprise: determining that the second image isoccluded by an occlusion, wherein the display image is displayed inresponse to determining that the second image is occluded by theocclusion. In some aspects, the methods, apparatuses, andcomputer-readable medium described above further comprise: determiningdepth information corresponding to one or more objects depicted in theimage and in the second image by processing the image and the secondimage.

In some aspects, the apparatus comprises a camera, a mobile device(e.g., a mobile telephone or so-called “smart phone” or other mobiledevice), a wireless communication device, a wearable device, an extendedreality device (e.g., a virtual reality (VR) device, an augmentedreality (AR) device, or a mixed reality (MR) device), a personalcomputer, a laptop computer, a server computer, or other device. In someaspects, the one or more processors include an image signal processor(ISP). In some aspects, the apparatus includes a camera or multiplecameras for capturing one or more images. In some aspects, the apparatusincludes an image sensor that captures the image data. In some aspects,the apparatus further includes a display for displaying the image, oneor more notifications associated with processing of the image, and/orother displayable data.

This summary is not intended to identify key or essential features ofthe claimed subject matter, nor is it intended to be used in isolationto determine the scope of the claimed subject matter. The subject mattershould be understood by reference to appropriate portions of the entirespecification of this patent, any or all drawings, and each claim.

The foregoing, together with other features and embodiments, will becomemore apparent upon referring to the following specification, claims, andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the present application are described indetail below with reference to the following figures:

FIG. 1 is a block diagram illustrating an example of an architecture ofan image capture and processing device;

FIG. 2A is an exploded front view of a mobile device with a display, anunder-display camera, and an under-display speaker;

FIG. 2B is a non-exploded front view of the mobile device of FIG. 2A;

FIG. 2C is a cutaway side view of the mobile device of FIG. 2A;

FIG. 2D is a front view of the mobile device of FIG. 2A with a firstunder-display camera, a second under-display camera, and anunder-display sensor;

FIG. 3A is a front view of a mobile device with an under-display camerain which the display displays a black bezel-style indicator over thecamera;

FIG. 3B is a front view of a mobile device with an under-display camerain which the display displays a black notch-shaped indicator over thecamera;

FIG. 3C is a front view of a mobile device with an under-display camerain which the display displays a black circular indicator over thecamera;

FIG. 4 is a front view of a mobile device with an under-display camerain which the display displays a triangular indicator around an areapositioned over the camera;

FIG. 5 is a front view of a mobile device with an under-display camerain which the display displays a circular indicator over the camera andan area selection user interface (UI) element to select a selectablearea over the camera;

FIG. 6 is a front view of a mobile device that compensates for occlusionof an under-display camera by a user’s hand during a touch, hover, orgesture input;

FIG. 7 is a front view of a mobile device with multiple under-displaycameras and under-display sensors in which the display displays aninterface area for controlling the multiple under-display cameras andunder-display sensors;

FIG. 8 is a flow diagram illustrating an example of an image processingtechnique;

FIG. 9 is a flow diagram illustrating another example of an imageprocessing technique; and

FIG. 10 is a diagram illustrating an example of a system forimplementing certain aspects of the present technology.

DETAILED DESCRIPTION

Certain aspects and embodiments of this disclosure are provided below.Some of these aspects and embodiments may be applied independently andsome of them may be applied in combination as would be apparent to thoseof skill in the art. In the following description, for the purposes ofexplanation, specific details are set forth in order to provide athorough understanding of embodiments of the application. However, itwill be apparent that various embodiments may be practiced without thesespecific details. The figures and description are not intended to berestrictive.

The ensuing description provides exemplary embodiments only, and is notintended to limit the scope, applicability, or configuration of thedisclosure. Rather, the ensuing description of the exemplary embodimentswill provide those skilled in the art with an enabling description forimplementing an exemplary embodiment. It should be understood thatvarious changes may be made in the function and arrangement of elementswithout departing from the spirit and scope of the application as setforth in the appended claims.

An image capture device (e.g., a camera) is a device that receives lightand captures image frames, such as still images or video frames, usingan image sensor. The terms “image,” “image frame,” and “frame” are usedinterchangeably herein. An image capture device typically includes atleast one lens that receives light from a scene and directs the lighttoward an image sensor of the image capture device. The light receivedby the lens can pass through an aperture controlled by one or morecontrol mechanisms. The light received by the lens is received by theimage sensor after passing through the lens. The one or more controlmechanisms can control exposure, focus, and/or zoom based on informationfrom the image sensor and/or based on information from an imageprocessor (e.g., a host or application process and/or an image signalprocessor). In some examples, the one or more control mechanisms includea motor or other control mechanism that moves a lens of an image capturedevice to a target lens position.

As described in more detail below, systems and techniques are describedherein for using one or more under-display cameras and/or one or moreunder-display sensors positioned underneath a display of a device (e.g.,a mobile device or other device).. Some image capture devices includeone or more front-facing cameras on a front surface of the image capturedevice. In some examples, the one or more front-facing cameras face thesame direction that a display screen of the image capture device faces.In videoconferencing as an illustrative example, the one or morefront-facing cameras capture images (e.g., video frames) of a user whilethe display screen faces the user and displays images (e.g., videoframes) of one or more other users connected to the videoconferenceand/or of the user himself/herself. A user can capture an image ofhimself or herself — referred to as a “selfie” — using the one or morefront-facing cameras that face the user. The user can align the captureof the image to frame and align the user’s depiction in the image byviewing a display image frames displayed on the display screen thatfaces the user while the user moves the image capture device intoposition for capturing the image. The display image frames may be basedon image frames that are captured by the one or more front-facingcameras and/or stored temporarily in an image buffer, and may thereforeact as a preview of the field(s) of view of the one or more front-facingcameras.

In some cases, the one or more front-facing cameras may be positionedadjacent to the display screen in a direction parallel to a planarsurface of the display screen. For instance, the one or morefront-facing cameras can be positioned in a bezel around the displayscreen. In some cases, a display screen can include a notch or a cutoutwithin which the display screen does not display anything. A cutout cansometimes be referred to as a “hole” or a “hole punch” in the displayscreen. Each of the one or more front-facing cameras can be positionedwithin such a bezel, notch, or cutout. One benefit of placing afront-facing camera in a bezel, notch, or cutout is that the userunderstands where the front-facing camera is positioned on the frontsurface of the image capture device. The bezel, notch, or cutout can insome cases be one of the only positions on a front surface of the devicethat is not covered by a display screen. Presence of a bezel, notch, orcutout on a front surface of a device can be seen as an indicator that afront-facing camera is positioned within the bezel, notch, or cutout.However, the presence of bezels, notches, and/or cutouts in a deviceprevents a display screen from occupying a full surface of an imagecapture device. Such a side effect of bezels, notches, and/or cutoutscan cause issues, especially on devices such as cellular phones thathave small front surfaces. Use of smaller display screens can result inaccessibility issues. For example, users with poor eyesight or visionproblems can have difficulty seeing visual elements that are displayedon the display screen. If the display screen is a touchscreen, userswith poor eyesight may have problems with seeing and successfullyinteracting with touch-based user interface elements displayed on thedisplay screen. Positioning of cameras in bezels, notches, and/orcutouts can also cause gaze mismatch issues. Gaze mismatch refers to theuser looking at the display rather than the camera during capture of animage of the user by the camera. This causes the user to be depictedwith their eyes looking off to the side in the captured image. This isfrequently a problem in videoconferencing, as the user generally looksat the display to see the second user that the user is speaking with,and in doing so, is not looking at the camera. Thus, neither userappears to be making eye contact with the other during thevideoconference.

A device may include a camera positioned underneath a display screen.Such a camera can be referred to as an under-display camera. Anunder-display camera can capture images using light that passes throughthe display screen underneath which the under-display camera ispositioned. In some examples, the under-display camera can capture animage using light that passes through the display screen while thedisplay screen is in use and displaying visual content. In someexamples, the under-display camera can capture an image using light thatpasses through the display screen while the display screen is turned offand not displaying any visual content. In some examples, theunder-display camera can capture an image using light that passesthrough the display screen during brief periods in which the displayscreen flickers off while the display screen is otherwise turned on anddisplaying visual content.

A device with one or more under-display cameras can include a displayscreen that faces the same direction as the one or more under-displaycameras and that occupies a full front surface of the device without anybezels, notches, or cutouts. However, devices with under-display camerascan have issues. For instance, if the camera is underneath the displayscreen, a user may not know where the camera is located. In one example,a user wishing to look at the front-facing camera while an image isbeing captured, as is often the case for selfie images or group images,may not know where on the device to look. A user interacting with thetouchscreen while the under-display camera is active may also obscurethe camera with the user’s finger or stylus by touching a portion of thetouchscreen that is positioned over the under-display camera. Forexample, a device can include a touchscreen user interface that allows auser to touch an area of an image to select that area of the image. Insome cases, a user may accidentally end up occluding the camera whiletouching the display to try to select an area of the image.

The issues with under-display cameras are not limited to front-facingdisplays and/or front-facing cameras. Some devices may have multipledisplays and/or multiple under-display cameras on different surfaces.For example, some devices may have a front surface and a rear surface,and may have displays with under-display cameras on the front surfaceand the rear surface. Some devices may have folding displays, or mayhave a fold or hinge between two separate displays. Those displays, andthe under-display cameras underneath each of them, may face differentdirections at different times. Some under-display cameras may beoccluded and disabled by the device when a foldable device is foldedclosed.

FIG. 1 is a block diagram illustrating an architecture of an imagecapture and processing system 100. The image capture and processingsystem 100 includes various components that are used to capture andprocess images of scenes (e.g., an image of a scene 110). The imagecapture and processing system 100 can capture standalone images (orphotographs) and/or can capture videos that include multiple images (orvideo frames) in a particular sequence. A lens 115 of the system 100faces a scene 110 and receives light from the scene 110. The lens 115directs the light toward the image sensor 130. The light received by thelens 115 passes through an aperture controlled by one or more controlmechanisms 120 and is received by an image sensor 130.

The one or more control mechanisms 120 may control exposure, focus,white balance, black balance, and/or zoom based on information from theimage sensor 130 and/or based on information from the image processor150. The one or more control mechanisms 120 may include multiplemechanisms and components; for instance, the control mechanisms 120 mayinclude one or more exposure control mechanisms 125A, one or more focuscontrol mechanisms 125B, and/or one or more zoom control mechanisms125C. The one or more control mechanisms 120 may also include additionalcontrol mechanisms besides those that are illustrated, such as controlmechanisms controlling analog gain, flash, HDR, depth of field, and/orother image capture properties.

The focus control mechanism 125B of the control mechanisms 120 canobtain a focus setting. In some examples, focus control mechanism 125Bstore the focus setting in a memory register. Based on the focussetting, the focus control mechanism 125B can adjust the position of thelens 115 relative to the position of the image sensor 130. For example,based on the focus setting, the focus control mechanism 125B can movethe lens 115 closer to the image sensor 130 or farther from the imagesensor 130 by actuating a motor or servo (or other lens mechanism),thereby adjusting focus. In some cases, additional lenses may beincluded in the system 100, such as one or more microlenses over eachphotodiode of the image sensor 130. Each of the microlenses can directthe light received from the lens 115 toward the corresponding photodiodebefore the light reaches the photodiode. The focus setting may bedetermined via contrast detection autofocus (CDAF), phase detectionautofocus (PDAF), hybrid autofocus (HAF), or some combination thereof.The focus setting may be determined using the control mechanism 120, theimage sensor 130, and/or the image processor 150. The focus setting maybe referred to as an image capture setting and/or an image processingsetting.

The exposure control mechanism 125A of the control mechanisms 120 canobtain an exposure setting. In some cases, the exposure controlmechanism 125A stores the exposure setting in a memory register. Basedon this exposure setting, the exposure control mechanism 125A cancontrol a size of the aperture (e.g., aperture size or f/stop), aduration of time for which the aperture is open (e.g., exposure time orshutter speed), a sensitivity of the image sensor 130 (e.g., ISO speedor film speed), analog gain applied by the image sensor 130, or anycombination thereof. The exposure setting may be referred to as an imagecapture setting and/or an image processing setting.

The zoom control mechanism 125C of the control mechanisms 120 can obtaina zoom setting. In some examples, the zoom control mechanism 125C storesthe zoom setting in a memory register. Based on the zoom setting, thezoom control mechanism 125C can control a focal length of an assembly oflens elements (lens assembly) that includes the lens 115 and one or moreadditional lenses. For example, the zoom control mechanism 125C cancontrol the focal length of the lens assembly by actuating one or moremotors or servos (or other lens mechanism) to move one or more of thelenses relative to one another. The zoom setting may be referred to asan image capture setting and/or an image processing setting. In someexamples, the lens assembly may include a parfocal zoom lens or avarifocal zoom lens. In some examples, the lens assembly may include afocusing lens (which can be lens 115 in some cases) that receives thelight from the scene 110 first, with the light then passing through anafocal zoom system between the focusing lens (e.g., lens 115) and theimage sensor 130 before the light reaches the image sensor 130. Theafocal zoom system may, in some cases, include two positive (e.g.,converging, convex) lenses of equal or similar focal length (e.g.,within a threshold difference of one another) with a negative (e.g.,diverging, concave) lens between them. In some cases, the zoom controlmechanism 125C moves one or more of the lenses in the afocal zoomsystem, such as the negative lens and one or both of the positivelenses.

The image sensor 130 includes one or more arrays of photodiodes or otherphotosensitive elements. Each photodiode measures an amount of lightthat eventually corresponds to a particular pixel in the image producedby the image sensor 130. In some cases, different photodiodes may becovered by different color filters, and may thus measure light matchingthe color of the filter covering the photodiode. For instance, Bayercolor filters include red color filters, blue color filters, and greencolor filters, with each pixel of the image generated based on red lightdata from at least one photodiode covered in a red color filter, bluelight data from at least one photodiode covered in a blue color filter,and green light data from at least one photodiode covered in a greencolor filter. Other types of color filters may use yellow, magenta,and/or cyan (also referred to as “emerald”) color filters instead of orin addition to red, blue, and/or green color filters. Some image sensors(e.g., image sensor 130) may lack color filters altogether, and mayinstead use different photodiodes throughout the pixel array (in somecases vertically stacked). The different photodiodes throughout thepixel array can have different spectral sensitivity curves, thereforeresponding to different wavelengths of light. Monochrome image sensorsmay also lack color filters and therefore lack color depth.

In some cases, the image sensor 130 may alternately or additionallyinclude opaque and/or reflective masks that block light from reachingcertain photodiodes, or portions of certain photodiodes, at certaintimes and/or from certain angles, which may be used for phase detectionautofocus (PDAF). The image sensor 130 may also include an analog gainamplifier to amplify the analog signals output by the photodiodes and/oran analog to digital converter (ADC) to convert the analog signalsoutput of the photodiodes (and/or amplified by the analog gainamplifier) into digital signals. In some cases, certain components orfunctions discussed with respect to one or more of the controlmechanisms 120 may be included instead or additionally in the imagesensor 130. The image sensor 130 may be a charge-coupled device (CCD)sensor, an electron-multiplying CCD (EMCCD) sensor, an active-pixelsensor (APS), a complimentary metal-oxide semiconductor (CMOS), anN-type metal-oxide semiconductor (NMOS), a hybrid CCD/CMOS sensor (e.g.,sCMOS), or some other combination thereof.

The image processor 150 may include one or more processors, such as oneor more image signal processors (ISPs) (including ISP 154), one or morehost processors (including host processor 152), and/or one or more ofany other type of processor 1010 discussed with respect to the computingsystem 1000. The host processor 152 can be a digital signal processor(DSP) and/or other type of processor. In some implementations, the imageprocessor 150 is a single integrated circuit or chip (e.g., referred toas a system-on-chip or SoC) that includes the host processor 152 and theISP 154. In some cases, the chip can also include one or moreinput/output ports (e.g., input/output (I/O) ports 156), centralprocessing units (CPUs), graphics processing units (GPUs), broadbandmodems (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components(e.g., Bluetooth®, Global Positioning System (GPS), etc.), anycombination thereof, and/or other components. The I/O ports 156 caninclude any suitable input/output ports or interface according to one ormore protocol or specification, such as an Inter-Integrated Circuit 2(I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a SerialPeripheral Interface (SPI) interface, a serial General PurposeInput/Output (GPIO) interface, a Mobile Industry Processor Interface(MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, anAdvanced High-performance Bus (AHB) bus, any combination thereof, and/orother input/output port. In one illustrative example, the host processor152 can communicate with the image sensor 130 using an I2C port, and theISP 154 can communicate with the image sensor 130 using an MIPI port.

The image processor 150 may perform a number of tasks, such asdemosaicing, color space conversion, image frame downsampling, pixelinterpolation, automatic exposure (AE) control, automatic gain control(AGC), CDAF, PDAF, automatic white balance, merging of image frames toform an HDR image, image recognition, object recognition, featurerecognition, receipt of inputs, managing outputs, managing memory, orsome combination thereof. The image processor 150 may store image framesand/or processed images in random access memory (RAM) 140/1020,read-only memory (ROM) 145/1025, a cache, a memory unit, another storagedevice, or some combination thereof.

Various input/output (I/O) devices 160 may be connected to the imageprocessor 150. The I/O devices 160 can include a display screen, akeyboard, a keypad, a touchscreen, a trackpad, a touch-sensitivesurface, a printer, any other output devices 1035, any other inputdevices 1045, or some combination thereof. In some cases, a caption maybe input into the image processing device 105B through a physicalkeyboard or keypad of the I/O devices 160, or through a virtual keyboardor keypad of a touchscreen of the I/O devices 160. The I/O 160 mayinclude one or more ports, jacks, or other connectors that enable awired connection between the system 100 and one or more peripheraldevices, over which the system 100 may receive data from the one or moreperipheral device and/or transmit data to the one or more peripheraldevices. The I/O 160 may include one or more wireless transceivers thatenable a wireless connection between the system 100 and one or moreperipheral devices, over which the system 100 may receive data from theone or more peripheral device and/or transmit data to the one or moreperipheral devices. The peripheral devices may include any of thepreviously-discussed types of I/O devices 160 and may themselves beconsidered I/O devices 160 once they are coupled to the ports, jacks,wireless transceivers, or other wired and/or wireless connectors.

In some cases, the image capture and processing system 100 may be asingle device. In some cases, the image capture and processing system100 may be two or more separate devices, including an image capturedevice 105A (e.g., a camera) and an image processing device 105B (e.g.,a computing device coupled to the camera). In some implementations, theimage capture device 105A and the image processing device 105B may becoupled together, for example via one or more wires, cables, or otherelectrical connectors, and/or wirelessly via one or more wirelesstransceivers. In some implementations, the image capture device 105A andthe image processing device 105B may be disconnected from one another.

As shown in FIG. 1 , a vertical dashed line divides the image captureand processing system 100 of FIG. 1 into two portions that represent theimage capture device 105A and the image processing device 105B,respectively. The image capture device 105A includes the lens 115,control mechanisms 120, and the image sensor 130. The image processingdevice 105B includes the image processor 150 (including the ISP 154 andthe host processor 152), the RAM 140, the ROM 145, and the I/O 160. Insome cases, certain components illustrated in the image capture device105A, such as the ISP 154 and/or the host processor 152, may be includedin the image capture device 105A.

The image capture and processing system 100 can include an electronicdevice, such as a mobile or stationary telephone handset (e.g.,smartphone, cellular telephone, or the like), a desktop computer, alaptop or notebook computer, a tablet computer, a set-top box, atelevision, a camera, a display device, a digital media player, a videogaming console, a video streaming device, an Internet Protocol (IP)camera, or any other suitable electronic device. In some examples, theimage capture and processing system 100 can include one or more wirelesstransceivers for wireless communications, such as cellular networkcommunications, 802.11 wi-fi communications, wireless local area network(WLAN) communications, or some combination thereof. In someimplementations, the image capture device 105A and the image processingdevice 105B can be different devices. For instance, the image capturedevice 105A can include a camera device and the image processing device105B can include a computing device, such as a mobile handset, a desktopcomputer, or other computing device.

While the image capture and processing system 100 is shown to includecertain components, one of ordinary skill will appreciate that the imagecapture and processing system 100 can include more components than thoseshown in FIG. 1 . The components of the image capture and processingsystem 100 can include software, hardware, or one or more combinationsof software and hardware. For example, in some implementations, thecomponents of the image capture and processing system 100 can includeand/or can be implemented using electronic circuits or other electronichardware, which can include one or more programmable electronic circuits(e.g., microprocessors, GPUs, DSPs, CPUs, and/or other suitableelectronic circuits), and/or can include and/or be implemented usingcomputer software, firmware, or any combination thereof, to perform thevarious operations described herein. The software and/or firmware caninclude one or more instructions stored on a computer-readable storagemedium and executable by one or more processors of the electronic deviceimplementing the image capture and processing system 100.

The I/O 156 and/or I/O 160 may include connections to a display. Theconnections to the display included in the I/O 156 and/or I/O 160 caninclude output connections by which the image processor 150 transfers animage captured by the image sensor 130 and processed using the imageprocessor 150 to the display and/or to a display controller. The displaycontroller receives the image and causes the display to display theimage.

In some implementations, the display can be a touchscreen. Theconnections to the display included in the I/O 156 and/or I/O 160 caninclude input connections, through which the image processor 150 canreceive inputs from the touchscreen and/or a touchscreen controller. Forinstance, the touchscreen can receive a touch input, and the touchscreencontroller can transmit information about the touch input to the imageprocessor 150. The touch input may be an instance in which a portion ofthe touchscreen detects a touch by a user’s finger, by a stylus, by apointer, by a pen, by a pencil, or by another input device. Theinformation about the touch input can include, for example, coordinatesof the touch input on the touchscreen. The touch input can be receivedat the touchscreen while the touchscreen is displaying an image. Theinformation about the touch input can identify a portion of an imagethat is being displayed on the touchscreen at the portion of thetouchscreen that received the touch input. The portion of the image caninclude, for example, an area spanning one or more pixels of the image.The touch input can be received at the touchscreen while the touchscreenis displaying a user interface (UI). The UI can include one or more UIelements, such as virtual buttons or sliders. The information about thetouch input can identify a portion of the UI (e.g., a UI element) thatwas displayed on the touchscreen at the portion of the touchscreen thatreceived the touch input. The touchscreen can be a resistivetouchscreen, a capacitive touchscreen, a surface acoustic wave (SAW)touchscreen, an infrared (IR) touchscreen, an optical imagingtouchscreen, a dispersive signal touchscreen, a piezoelectrictouchscreen, an acoustic pulse recognition touchscreen, another type oftouchscreen discussed herein, or some combination thereof.

In some implementations, the display may include one or more other inputcomponents. For example, the display may include one or more activedepth sensors next to or underneath the display. The active depthsensors may face the same direction as the display. The active depthsensors may include stereo cameras. The active depth sensors may be anytype of sensors discussed herein with respect to the sensor 260. Theconnections to the display included in the I/O 156 and/or I/O 160 caninclude input connections, through which the image processor 150 canreceive inputs from the active depth sensors. Inputs from the activedepth sensors may be processed by the image processor 150 or anotherprocessor to determine distance from the sensors to an object, which maybe referred to as depth. If the active depth sensors include cameras,this depth determination may be performed using stereoscopic imageprocessing as discussed further herein. A device with the active depthsensors may thus determine three-dimensional coordinates of objects inthe environment around the device. The image processor 150 (or anotherprocessor) may process the sensor measurements from the active depthsensors to determine whether a user’s hand, finger, or pointing device(e.g., stylus) is hovering over, pointing at, gesturing toward, orgesturing around a user interface (UI) element displayed on the display.If the image processor 150 (or another processor) detects that the user(or pointing device) is hovering over, pointing at, gesturing toward, orgesturing around a particular UI element, the image capture andprocessing system 100 may trigger a function corresponding to that UIelement. Hover-based, pointing-based, or gesture-based interaction witha display may be more hygienic than touchscreen-based interaction withthe display, especially for image capture and processing systems 100that many different people are expected to interact with. For instance,these non-touch inputs may be more hygienic for image capture andprocessing systems 100 that are included in point of sale (POS)terminals, automated teller machines (ATM), restaurant ordering kiosks,information kiosks, or other kiosks.

At least a subset of the image capture and processing system 100 canpositioned underneath the display. In some examples, at least the imagesensor 130 and the lens 115 are positioned under the display. In oneexample, the image capture device 105A is positioned under the display.In some examples, the image processing device 105B is positioned underthe display. The term “under-display camera” may refer to at least asubset of an image capture and processing system 100 that is at leastpartially positioned underneath a display, at least a subset of an imagecapture device 105A that is at least partially positioned underneath adisplay, at least a subset of an image processing device 105B that is atleast partially positioned underneath a display, or some combinationthereof. A component or device that is positioned underneath a displaymay be referred to as an under-display component or device.

FIG. 2A is an exploded front view of a mobile device 205 with a display210, an under-display camera 220, and an under-display speaker 225. Alegend 290 is included in FIG. 2A that displays three axes representingdifferent spatial dimensions in FIG. 2A. These axes are identified asthe X axis, the Y axis, and the Z axis. The X axis and the Y axis areparallel to the planar surface of the display 210. The Z axis isperpendicular to the planar surface of the display 210. The mobiledevice 205 is illustrated with a body 215 that includes a front surface255, which is white in FIG. 2A. The body 215 also includes a housing250, which is shaded using diagonal stripes in FIG. 2A. As indicated bydashed lines and dashed arrows, the display 210 is in front of the frontsurface 255 of the body 215 along the Z axis, and is to be placed ontothe front surface 255 of the body 215. The front surface 255 of the body215 includes a camera 220 and a speaker 225. Placing the display 210onto the front surface of the body 215 includes placing the display 210onto and/or onto the camera 220 and the speaker 225 along the Z axis.Placing the display 210 onto the front surface of the body 215 coversthe camera 220 and the speaker 225 with the surface of the display 210along the Z axis.

The body 215 also includes a processor 270, a memory 275, and a battery280. The camera 220 and speaker 225 are connected to the processor 270and to the memory 275 via one or more data connection lines. In somecases, the processor 270 can include the image processor 150, the ISP154, the host processor 152, or some combination thereof. The processor270 can process image data captured by the image sensor of the camera220. The memory 275 can store images captured by the camera 220 and/orprocessed by the processor 270. The memory 275 can also store audio tobe output by the speaker 225. The processor 270 can process audio to beoutput by the speaker 225. The camera 220, the speaker 225, theprocessor 270 and or/ the memory 275 receive power from the battery 280over one or more data lines.

The camera 220 can be any type of camera discussed herein. For instance,the camera 220 may include at least a subset of an image capture device105A, at least a subset of an image processing device 105B, at least asubset of an image capture and processing system 100, at least a subsetof any other camera device or system discussed herein, or a combinationthereof. The camera 220 can use any type of lens. For instance, thecamera can use a normal lens, a wide-angle lens, a fisheye lens, along-focus lens, a telephoto lens, a zoom lens, or a combinationthereof. The camera 220 can detects light within a particular spectrumof light. The spectrum of light may be a subset of the electromagneticspectrum. For example, the spectrum of light may be, or may include, thevisible light (VL) spectrum, the infrared (IR) light spectrum, the nearinfrared (NIR) light spectrum, the ultraviolet (UV) light spectrum,another subset of the electromagnetic spectrum, or a combinationthereof. The NIR light spectrum may refer to a subset of the IR lightspectrum that is adjacent to the VL spectrum. The image sensor of thecamera 220 may be a complementary metal-oxide-semiconductor (CMOS) imagesensor, a charge-coupled device (CCD) image sensor, another type ofimage sensor, or a combination thereof. The image sensor of the camera220 may be any size of image sensor.

FIG. 2B is a non-exploded front view of the mobile device 205 of FIG.2A. The camera 220 is positioned underneath the display 210 and iscovered by the display 210 in FIG. 2B. The legend 290 of FIG. 2A is alsoincluded in FIG. 2B. The camera 220 may be referred to as anunder-display camera. The speaker 225 is positioned underneath thedisplay 210 along the Z axis, and is covered by the display 210 in FIG.2B. The speaker 225 may be referred to as an under-display speaker. Thespeaker 225 is positioned near the top of the mobile device 205 alongthe Y axis, and may output audio toward the display 210, so that theaudio waves travel through the display 210. A user holding the mobiledevice 205 adjacent to his or her ear can hear the audio output by thespeaker 225 through the display 210. The speaker 225 and the camera 220may be adjacent to the display 210 along the Z axis. Because the camera220 is entirely covered by the display 210, any incident light thatreaches the camera 220 passes through a light-passing area 285 of thedisplay 210 before reaching the camera 220. The light-passing area 285is illustrated using dashed lines, and at least as large as the lens ofthe camera 220. In some devices, a camera may be only partly covered bya display 210, in which case some of the incident light that reachessuch a camera may pass through the area 285 of the display 210, whileother incident light that reaches such a camera may pass through anotherdisplay, a non-display surface, or no surface at all (e.g., air oranother fluid). In some cases, the light-passing area 285 of the display210 may simply be referred to as a portion of the display 210, or as anarea of the display 210.

For simplicity, the processor 270, memory 275, and the battery 280 arenot illustrated in FIG. 2B. Likewise, the data and power lines betweenthe camera 220, the speaker 225, the processor 270, the memory 275, andthe battery 280 are not illustrated in FIG. 2B. It should be understoodthat these components can still be present in the mobile device 205 ofFIG. 2B in a similar configuration to the configuration illustrated inFIG. 2A, positioned underneath the display 210 along the Z axis.

The display 210 is illustrated as being slightly smaller than the frontsurface 255 of the body 215 of the mobile device 205 so that the display210 is clearly visible within FIG. 2A and FIG. 2B as a distinctcomponent of the mobile device 205. The mobile device 205 thus includesa small bezel around the display 210. However, this bezel need not bepresent in the mobile device 205. In some cases, the display 210 cancover the full vertical length and/or the full horizontal length of thefront surface 255 of the body 215 of the mobile device 205, from edge toedge, with no bezel around the display 210. In some cases, the display210 can include one or more portions that curve around an edge of thefront surface 255 of the body 215 of the mobile device 205, so that thedisplay 210 may even wrap around to at least a portion of the sides ofthe mobile device 210 that connect the front surface 255 of the body 215to the rear surface 240 of the body 215 of the mobile device 205.

FIG. 2C is a cutaway side view of the mobile device 205 of FIG. 2A. Alegend 295 showing only the Y axis and the Z axis of the legend 290 isincluded in FIG. 2C due to the change in perspective from FIG. 2A andFIG. 2B to FIG. 2C. In the side view of the mobile device 205 of FIG.2C, the front surface 252 of the display 210 appears on the left side ofthe mobile device 205, while the rear surface 240 of the body 215appears on the right side of the mobile device 205. The body 215includes a housing 250 that is illustrated as a region shaded withhorizontal stripes. The body 215 also includes the camera 220 and thespeaker 225. A portion of the body 215 may be coupled to the camera 220and/or to the speaker 225. The front portions of the camera 220 and thespeaker 225 form parts of the front surface 255 of the body 215. Thecamera 220 and the speaker 225 are illustrated as adjacent to thedisplay 210 and/or coupled to the display 210 along the Z axis. In somecases, there may be some space along the Z axis between the camera 220and the display 210, and/or between the speaker 225 and the display 210.

Several rays of light 245 are illustrated as arrows passing through thedisplay 210 and reaching the camera 220. An image sensor 130 of thecamera 220 receives the light 245 that passes through at least a portionof the display 210. In some examples, the display 210 may includemicrolenses in interstitial regions between the pixels of the display.The display 210 may be a touchscreen. In some aspects, the interstitialregions, and therefore the microlenses, are also positioned betweentraces of a touch-sensitive touchscreen surface of the display 210.These traces may be capacitive traces, resistive traces, or othertraces, depending on the type of touchscreen. In some examples, thelight 245 passes through transparent and/or translucent layers of liquidcrystal, glass, and/or plastic of the display 210 before reaching theunder-display camera. In some cases, the cameras 220 and/or associatedprocessor 270 can perform image processing to compensate for colorbiasing from color associated with the materials of the display 210 thatthe light 245 passes through before reaching the camera 220, such ascolor filters. The light 245 received by the image sensor 130 of thecamera 220 may be incident light that passes through at least a portionof the display 210 before reaching the camera 220 (e.g., beforeimpinging an image sensor 130 of the camera 220).

Several audio waves 260 are illustrated as arrows passing through thedisplay 210. The audio waves 260 continue past the front surface 252 ofthe display 210 and propagate into the air. The audio waves 260 can beheard by a user when adjacent to or within a range of the mobile device205.

For simplicity, the processor 270, memory 275, and the battery 280 arenot illustrated in FIG. 2C. Likewise, the data and power lines betweenthe camera 220, the speaker 225, the processor 270, the memory 275, andthe battery 280 are not illustrated in FIG. 2C. It should be understoodthat these components can still be present in the mobile device 205 ofFIG. 2C in a similar configuration to the configuration illustrated inFIG. 2A.

FIG. 2D is a front view of the mobile device 205 of FIG. 2A with a firstunder-display camera 220A, a second under-display camera 220B, and anunder-display sensor 260. The first under-display camera 220A and thesecond under-display camera 220B may be referred to as the first camera220A and the second camera 220B, respectively. The under-display sensor260 may be referred to as the sensor 260. The legend 290 of FIG. 2A isalso included in FIG. 2D. The mobile device 205 is an example of adevice with multiple under-display cameras. Like the camera 220 of FIGS.2A-2C, the first camera 220A and the second camera 220B are positionedunderneath the display 210 along the Z axis. In the context of FIG. 2C,the first camera 220A and the second camera 220B may be positionedbetween the display 210 and the housing 250 of the body 215 of themobile device 205, similarly to the camera 220 and the speaker 225 ofFIG. 2C. The first camera 220A can receive incident light that passesthrough at least a first portion of the display 210 before reaching thefirst camera 220A. The second camera 220B can receive incident lightthat passes through at least a second portion of the display 210 beforereaching the second camera 220B. Incident light reaching a camera caninclude the incident light impinging the image sensor of the camera. Thefirst camera 220A and the second camera 220B can capture images based onreceipt of this incident light at their respective image sensors. Thefirst camera 220A is closer to a first side of the mobile device 205,while the second camera 220B is closer to a second side of the mobiledevice 205. In the example mobile device 205 illustrated in FIG. 2D, thefirst side is the top of the mobile device 205, while the second side isthe bottom of the bottom of the mobile device. In other examples, thefirst side and/or the second side may be the left side or the rightside.

Because the first camera 220A and the second camera 220B are positioneda distance apart, the first camera 220A and the second camera 220B canbe used as stereoscopic cameras. The first camera 220A can capture afirst image and the second camera 220B can capture a second image withinthe same time window. The time window may be a predetermined duration oftime, such as one or more picoseconds, one or more nanoseconds, one ormore milliseconds, one or more seconds, or a combination thereof. Themobile device 205, or another device that receives the first image andthe second image from the mobile device 205, can process the first imageand the second image using stereoscopic image processing to determinedepth information corresponding to the first image and the second image.Depth information can refer to a depth of different objects depicted inthe first image and in the second image. Depth can refer to a distancebetween a photographed object and the mobile device 205. The greater thedistance between the photographed object and the mobile device 205, thegreater the depth of the object. The lower the distance between thephotographed object and the mobile device 205, the lower the depth ofthe object. Stereoscopic image processing may also be referred to asstereo image processing, multi image processing, binocular imageprocessing, stereoscopic camera processing, stereo camera processing,multi camera processing, binocular camera processing, or a combinationthereof.

Stereoscopic image processing can use stereo disparity, binoculardisparity, stereopsis, stereo triangulation, triangulation,stereophotogrammetry, photogrammetry, three-dimensional (3D) mapping, 3Dmodeling, or a combination thereof to determine depth informationcorresponding to at least a subset of pixels of the first image and/orto at least a subset of pixels of the second image. In some examples,stereoscopic image processing can include detecting a first depiction ofa feature (e.g., a corner or edge) in the first image, and detecting asecond depiction of the same feature in the second image. Stereoscopicimage processing can determine a depth of the feature by determining adistance between a position of the first depiction of the feature in thefirst image and a position of the second depiction of the feature in thesecond image. This distance can be referred to as a stereo disparity,stereoscopic disparity, or binocular disparity. The greater the stereodisparity, the lower the depth of the feature, and the closer thefeature is to the mobile device 205. The lower the stereo disparity, thehigher the depth of the feature, and the farther the feature is from themobile device 205.

For example, the first image and the second image may both depict aperson in front of a background, similarly to the display image 350displayed on the display 210 of FIG. 3A. The mobile device 205 canprocess the first image and the second image using stereoscopic imageprocessing to identify that the pixels depicting the person correspondto a low depth value because the person is close to the mobile device205. The mobile device 205 can process the first image and the secondimage using stereoscopic image processing to identify that the pixelsdepicting the background correspond to a high depth value because thebackground is far from the mobile device 205.

The mobile device 205 may include more than two under-display camerasunder the display 210. For instance, the mobile device 205 may includethree under-display cameras, four under-display cameras, fiveunder-display cameras, six under-display cameras, seven under-displaycameras, eight under-display cameras, nine under-display cameras, tenunder-display cameras, or more than ten under-display cameras. A mobiledevice 205 with N under-display cameras under the display 210 cancapture at least N images of a scene within the same time window. Eachof the N images of the scene can be captured with a different one of theN under-display cameras under the display 210. All N images can beprocessed using stereoscopic image processing, or multi imageprocessing, to determine depth information for pixels and/or featuresdepicted in at least a subset of the N images. If N is greater than two,the depth information determined can be even more accurate and detailedthan if N is two.

In some examples, the first camera 220A and the second camera 220B shareat least one component, type of component, feature, functionality, or acombination thereof. For example, the first camera 220A can use the sametype of lens as the second camera 220B, the first camera 220A can detectlight within the same spectrum of light as the second camera 220B, thefirst camera 220A can use the same type of image sensor as the secondcamera 220B, the first camera 220A can use the same dimensions of imagesensor as the second camera 220B, or a combination thereof.

In some examples, the first camera 220A differs from the second camera220B through the use of at least one different component, type ofcomponent, feature, functionality, or a combination thereof. Forexample, the first camera 220A can use a first lens, while the secondcamera 220B uses a second lens. The first lens may have a wider anglethan the second lens, for instance if the first lens is a wide-anglelens or a fisheye lens. The wider angle of the first lens may grant thefirst camera 220A a wider field of view than the second camera 220B. Thefirst lens may be a telephoto lens, while the second lens is not atelephoto lens, providing the first camera 220A with a longer focus thanthe second camera 220B. The first lens may be a zoom lens, while thesecond lens is not a zoom lens, providing the first camera 220A with alarger zoom than the second camera 220B. In some examples, the imagesensor of the first camera 220A detects light within a first spectrum oflight, while the image sensor of the second camera 220B detects lightwithin a second spectrum of light. Each spectrum of light may refer to adifferent subset of the full electromagnetic spectrum. For example, thefirst spectrum of light can be the visible light (VL) spectrum, whilethe second spectrum of light can be the infrared (IR) light spectrum.The second spectrum of light may be the near infrared (NIR) lightspectrum, which is a subset of the IR light spectrum adjacent to the VLlight spectrum. The second spectrum of light may be the ultraviolet (UV)light spectrum. In some examples, the first image sensor of the firstcamera 220A can be a different type of image sensor than the secondimage sensor of the second camera 220B. For example, the first imagesensor may be a complementary metal-oxide-semiconductor (CMOS) imagesensor, while the second image sensor is a charge-coupled device (CCD)image sensor. The dimensions of the first image sensor may be differentfrom the dimensions of the second image sensor, so that one is largerthan the other. The first camera 220A and the second camera 220B candiffer in any combination of the ways discussed herein.

The sensor 260 can be any type of sensor. For example, the sensor 260can include at least one of a radio detection and ranging (RADAR)sensor, a light detection and ranging (LIDAR) sensor, an electromagneticdetection and ranging (EMDAR) sensor, a multispectral imaging detectionand active reflectance (MIDAR) sensor, a sound detection and ranging(SODAR) sensor, a sound navigation and ranging (SONAR) sensor, a laserrangefinder, a time-of-flight (TOF) emitter/sensor, a structured lightemitter/sensor, a camera with a coded aperture, an active depth sensor,a range sensor, a microphone, an accelerometer, a gyroscope, a globalnavigation satellite system (GNSS) receiver, a global positioning system(GPS) receiver, another sensor discussed herein, or a combinationthereof. Like the camera 220 of FIGS. 2A-2C, the sensor 260 ispositioned underneath the display 210. In the context of FIG. 2C, thesensor 260 may be positioned between the display 210 and the housing 250of the body 215 of the mobile device 205, similarly to the camera 220and the speaker 225 of FIG. 2C. The sensor 260 can receive incidentlight, sound, radio waves, microwaves, IR waves (e.g., NIR waves), UVwaves, electromagnetic (EM) radiation, and/or other signals that passthrough at least a portion of the display 210 before impinging thesensor 260. In some cases, sensor 260 can output light, sound,electromagnetic (EM) radiation, and/or other signals that pass throughat least a portion of the display 210 and toward the environment afterbeing output by the sensor 260. In some cases, the signals output by thesensor 260 can reflect off of objects in the environment, bounce backtoward the sensor 260, and be received by the sensor 260.

FIG. 3A is a front view of a mobile device 205 with an under-displaycamera 220 in which the display 210 displays a black bezel-styleindicator 310A in an indicator area. The term “indicator area” may referto an area within which an indicator (such as the black bezel-styleindicator 310A or another indicator) is displayed. An indicator area mayinclude at least a subset of the light-passing area 290 over the camera220. The legend 290 of FIG. 2A is also included in FIG. 3A. The camera220 of the mobile device 205 captures one or more images. In someexamples, the camera 220 can capture images continually, and maytemporarily store those images in an image buffer. The mobile device 205is illustrated displaying a display image 350 on the display 210. Insome examples, the display image 350 is one of the images captured bythe camera 220 and/or stored in the image buffer on the display 210. Insome examples, the display image 350 is based on one of the imagescaptured by the camera 220 and/or stored in the image buffer on thedisplay 210. In some examples, the display image 350 is a modifiedvariant of one of the images captured by the camera 220 and/or stored inthe image buffer on the display 210. For example, the mobile device 205may generate the display image 350 by modifying an image captured by thecamera 220 using cropping, upscaling, downscaling, stretching, skewing,distorting, warping, adjusting brightness, adjusting contrast, adjustingcolor saturation, adjusting tone mapping, adjusting white balance,adjusting black balance, adjusting a histogram, adjusting levels,correcting red eye, overlaying text, overlaying user interfaceelement(s), overlaying graphics, overlaying another image, overlayingelement(s) of another image, merging the image with another image (e.g.,to generate a HDR image), any other image processing image modificationdiscussed herein, or a combination thereof. In one example, the displayimage 350 may be a version of an image captured by the camera 220 thatis scaled so that a majority of the image is visible on the display 210based on the resolution of the display 210, that is cropped to fit thedimensions of the display 210, and whose brightness and contrast andsaturation are adjusted to display well on the display 210. The displayimage 350 can be used as a preview image previewing the camera 220′sfield of view by previewing a display image 350 that is based on animage that the camera 220 recently captured (e.g., an image that thecamera 220 most recently captured). The display image 350 displayed onthe display 210 in FIG. 3A is illustrated as a selfie image of a user infront of a background. The background includes buildings, trees, grass,and other foliage.

Because the camera 220 is underneath the display 210, a user of themobile device 205 may not know where the camera 220 is located withrespect to the front surface 255 of the display 210. A user wishing tolook at the camera while an image is being captured - for example duringcapture of the selfie image displayed on the display 210 as the displayimage 350 in FIG. 3A - may not know where to look. To resolve thisissue, the mobile device 205 of FIG. 3A displays a bezel-shapedindicator 310A overlaid over the display image 350 within the indicatorarea of the display 210. The bezel-shaped indicator 310 has a roundedrectangular shape outlining an area of the image where the camera 220 ispositioned under the display 210 along the z-axis while the displayimage 350 is displayed on the display 210. The indicator 310 isillustrated as an area that is shaded in black. The mobile device 205displays the indicator 310A by causing a plurality of pixels that aregrouped together in the indicator area to display black. Depending onthe type of display 210, the display 210 can instruct pixels to displayblack by turning off power to those pixels. A user of the mobile device205 wanting to look at the camera 220 knows to look in the indicatorarea within which the bezel-style indicator 310A is displayed. Therounded rectangular shape of the indicator 310A of FIG. 3A mimics abezel. Because the pixels of the display 210 display black in theindicator 310A, any interference on the light 245 reaching the camera220 after passing through the display 210 is minimized.

The indicator 310A is large enough that a user might not understandwhich part of the indicator 310A rests atop the camera 220 along the Zaxis. To help the user identify the position of the camera 220, theindicator area may include additional indicators identifying theposition of the camera 220 underneath the display 210 may be overlaidatop the indicator 310A on the display 210. For example, two indicatorarrows 340 that point toward the camera 220 are illustrated as beingdisplayed overlaid atop the bezel-style indicator 310A in the indicatorarea on the display 210 in FIG. 3A. The indicator arrows may bedisplayed in a different color than the rest of the indicator area 310.Other types of additional indicators may be overlaid atop the indicator310A or otherwise within the indicator area on the display 210, such asan indicator shape. The indicator shape may outline a smaller area thanthe indicator 310A within the indicator area. For example, the indicatorshape may outline the position of the camera 220 positioned underneaththe smaller area along the Z-axis. Examples of indicator shapes includethe indicator 410 of FIG. 4 and the indicator 510 of FIG. 5 .

The mobile device 205 includes an image capture hardware button 330. Themobile device 205 may receive an image capture input when the imagecapture hardware button 330 is pressed. In some cases, the image capturehardware button 330 can be replaced and/or supplemented with anotherhardware user interface that receives the image capture input, such as aswitch, a touch-sensitive surface separate from the display 210, afingerprint sensor, another type of biometric scanner, a microphone, awheel, a knob, a slider, an accelerometer, a gyroscope, or somecombination thereof. The mobile device 205 also includes an imagecapture user interface (UI) element 320 that is displayed on a UI areaof the display 210, overlaid over the display image 350. The mobiledevice 205 may receive an image capture input when the touch-sensitivetouchscreen surface of the display 210 receives a touch area at the UIarea of the display 210.

In some examples, receipt of the image capture input trigger the camera220 of the mobile device 205 to capture a second image, and for themobile device 205 to receive and store the second image from the camera220. In some examples, receipt of the image capture input triggerstransfer of an image that was already captured by the camera 220 of themobile device 205 from temporary storage in an image buffer to long-termnontemporary storage in a memory 175.

FIG. 3B is a front view of a mobile device 205 with an under-displaycamera 220 in which the display 210 displays a black notch-shapedindicator 310B over the camera 220. The mobile device 205 displays theblack notch-shaped indicator 310B in the indicator area that is over thecamera 220 in the Z-axis. The legend 290 of FIG. 2A is also included inFIG. 3B. The mobile device 205 of FIG. 3B is similar to the mobiledevice 205 of FIG. 3A. The bezel-shaped indicator 310A of FIG. 3A isreplaced by a notch-shaped indicator 310B in FIG. 3B. The notch-shapedindicator 310B can occupy less of the display 210 than the bezel-shapedindicator 310A, allowing the display image 350 to fill up more of thedisplay 210.

FIG. 3C is a front view of a mobile device 205 with an under-displaycamera 220 in which the display 210 displays a black circular indicator310C over the camera 220. The mobile device 205 displays the blackcircular indicator 310C in the indicator area that is over the camera220 in the Z-axis The legend 290 of FIG. 2A is also included in FIG. 3C.The mobile device 205 of FIG. 3C is similar to the mobile device 205 ofFIG. 3A and FIG. 3B. The bezel-shaped indicator 310A of FIG. 3A isreplaced by a circular indicator 310C in FIG. 3C. The circular indicator310C can occupy less of the display 210 than the bezel-shaped indicator310A or the notch-shaped indicator 310B, allowing the display image 350to fill up more of the display 210.

The indicators 310A, 310B, and 310C are illustrated in FIG. 3A, FIG. 3B,and FIG. 3C as being filled in with black pixels or pixels that areturned off. In some examples, indicators 310A, 310B, and/or 310C can befilled in with pixels of a different color, such as white, grey, red,green, blue, yellow, or some mixture of these colors. The color that theindicators 310A, 310B, and/or 310C are filled in with can have any hue,any luminosity, and any opacity. In some examples, the indicators 310A,310B, and 310C may be filled in with a pattern, a texture, a gradient,or even another image distinct from the display image 350 that theindicators 310A, 310B, and 310C are overlaid over. In some examples, theindicators 310A, 310B, and 310C may be at least partially translucent ortransparent.

In some examples, the indicators 310A, 310B, and 310C can be overlaidover the display image 350 consistently over a period of time, such as aperiod of time during which a photo capture software application isrunning on the mobile device 205 and/or a period of time during whichthe under-display camera 220 is activated. In some examples, theindicators 310A, 310B, and 310C can blink on and off periodically duringthe period of time. In some examples, the indicators 310A, 310B, and310C fade in and out periodically during the period of time. In someexamples, the indicator can be displayed, called out, emphasized,hidden, or some combination thereof in response to a user input. Theuser input includes at least one of a tap on the display, a hover overthe display, a preconfigured gesture across the display, a user’spreference setting, or a voice input from the user. In some examples, atleast a subset of an indicator 310A, 310B, or 310C is animated, forexample to include a border that periodically expands, contracts,pulsates, or some combination thereof.

In some examples, the mobile device 205 displays the indicators 310A,310B, and/or 310C overlaid over the display image 350 by generating thedisplay image 350 so that it includes the indicators 310A, 310B, and/or310C. The camera 220 may capture an image, which the mobile device 205may modify as discussed previously to generate the display image 350.The mobile device 205 may further modify the display image 350 to mergethe display image 350 with the indicators 310A, 310B, and/or 310C. Insome examples, the indicators 310A, 310B, and/or 310C are overlaid overthe display image 350, but the display image 350 remains separate fromthe indicators 310A, 310B, and/or 310C. In some examples, the mobiledevice 205 overlays the indicators 310A, 310B, and/or 310C over thedisplay image 350 using a blending effect. Blending effects may include,for example, a dissolve effect, a multiply effect, a darkening effect, acolor burn effect, a linear burn effect, a lighten effect, a screeneffect, a color dodge effect, a linear dodge effect, an overlay effect,a soft light effect, a hard light effect, a vivid light effect, a pinlight effect, a hard mix effect, a difference effect, an exclusioneffect, a subtract effect, a divide effect, a threshold effect, a hueblend effect, a saturation blend effect, a color blend effect, aluminosity blend effect, a lighter color comparison effect, a darkercolor comparison effect, or some combination thereof.

FIG. 4 is a front view of a mobile device 205 with an under-displaycamera 220 in which the display 210 displays a triangular indicator 410around an area positioned over the camera 220. The legend 290 of FIG. 2Ais also included in FIG. 4 . Like the mobile device 205 of FIG. 3A, FIG.3B, and FIG. 3C, the mobile device 205 of FIG. 4 is illustrateddisplaying a display image 350 using its display 210. The display image350 may be based on an image captured by the camera 220. The displayimage 350 may be displayed on the display 210 temporarily as a previewimage to indicate to the user of the mobile device 205 how the scenethat the camera 220 is receiving light from is framed in the field ofview of the camera 220. The mobile device 205 displays the indicator 410overlaid over the display image 350. The indicator 410 includes anindicator shape that outlines an area positioned over the camera 220.The example indicator 410 illustrated in FIG. 4 is triangular. Theindicator 410 can be any shape. For instance, the indicator 410 can be acircle, a semicircle, a square, a triangle, a rectangle, an oval, atrapezoid, a parallelogram, a rhombus, a kite, a quadrilateral, apentagon, a hexagon, a heptagon, an octagon, an enneagon, a decagon, apolygon with more than 10 sides, or another shape.

In addition to including a area outline of a triangular shape, theindicator 410 is illustrated in FIG. 4 filled in with a shaded pattern.This shaded pattern may represent a shaded area or otherwise filled-inarea within the indicator shape of the indicator 410, similar to blacksharding of the indicators 310A, 310B, and 310C. The shaded area withinthe outlined indicator shape of the indicator 410 may have anycombination of the characteristics and traits discussed with respect tothe indicators 310A, 310B, and 310C. For example, the shaded area withinthe outlined indicator shape of the indicator 410 may be patterned andtranslucent. In some cases, the outlined indicator shape of theindicator 410 serves as the border or edge of the shaded area. In somecases, the shaded area of the indicator 410 and the outlined indicatorshape of the indicator 410 cover overlapping but distinct areas. In someexamples, the shaded area may be removed from the indicator 410, or mayblink on and off periodically. The indicator 410 can be the exclusiveindicator of the position of the camera 220 underneath the display 210,or can be used in addition to other indicators such as one of theindicators 310A-C and/or indicator arrows 340.

The lines that make up the outlined indicator shape of the indicator 410can be of any color, hue, luminosity, transparency, and/or thickness.The lines that make up the outlined indicator shape of the indicator 410can be a solid color, can be patterned, can include a gradient, and/orcan make up an image distinct from the display image 350 that theoutlined indicator shape of the indicator 410 is overlaid over. Thelines that make up the outlined indicator shape of the indicator 410 caninclude solid lines, dashed lines, dotted lines, dashdotted lines,dot-dashed-lines, any other type of line, or a combination thereof. Thelines that make up the outlined indicator shape of the indicator 410 caninclude single lines, double lines, triple lines, quadruple lines, morethan four lines, or a combination thereof. In some examples, theoutlined indicator shape of the indicator 410 is be overlaid over thedisplay image 350 consistently over a period of time, such as a periodof time during which a photo capture software application is running onthe mobile device 205 and/or a period of time during which theunder-display camera 220 is activated. In some examples, the outlinedindicator shape of the indicator 410 blink on and off periodicallyduring the period of time. In some examples, the outlined indicatorshape of the indicator 410 fades in and out periodically during theperiod of time. In some examples, at least a subset of the outlinedindicator shape of the indicator 410 is animated, for exampleperiodically expanding, contracting, pulsating, or some combinationthereof.

In some examples, the mobile device 205 displays the indicator 410overlaid over the display image 350 by modifying the display image 350to include the indicator 410. The modified display image may be a copyof the image merged with the indicator 410. In some examples, mobiledevice 205 blends the indicator 410 with the display image 350 that itoverlays using a blending effect, such as any of the blending effectsdiscussed with respect to the indicator areas 310A, 310B, and 310C.

The mobile device 205 of FIG. 4 also includes an image capture UIelement 320 and an image capture hardware button 330. These elements mayfunction similarly to the image capture UI element 320 and the imagecapture hardware button 330 of FIG. 3A, FIG. 3B, and FIG. 3C.

In some cases, the indicator 410, or any such indicator, may permanentlybe overlaid over the display image 350 on the display at an area of thedisplay through which the camera 220 receives incident light. In somecases, the indicator 410, or any such indicator, may be hidden bydefault, and may be overlaid over the display image 350 on the displayin response to receipt of an input. The input may be a touch of atouch-sensitive touchscreen surface of the display, a predefined gestureacross the touch-sensitive touchscreen surface of the display, a hoverover the display, a predefined gesture over the display, a voice inputreciting a predefined phrase recorded by a microphone. For example, theuser of the mobile device 205 may touch the touch-sensitive touchscreensurface of the display 210, triggering the mobile device 205 to displaythe indicator 410. The user may speak the phrase “where is my camera” ora similar phrase, which may act as a voice input. A microphone of themobile device 205 (or coupled to the mobile device 205) may record theuser saying this phrase as the voice input, and the mobile device 205(or a remote voice-to-text interpretation server) may recognize that thevoice input is saying a predefined phrase that triggers display of theindicator 410, triggering the mobile device 205 to display the indicator410.

FIG. 5 is a front view of a mobile device 205 with an under-displaycamera 220 in which the display 210 displays a circular indicator 510over the camera 220 and an area selection user interface (UI) element520 to select a selectable area 540 over the camera 220. The legend 290of FIG. 2A is also included in FIG. 5 . Like the mobile device 205 ofFIG. 3A, FIG. 3B, FIG. 3C, and FIG. 4 , the mobile device 205 of FIG. 5is illustrated displaying an display image 350 using its display 210.The display image 350 may be based on an image captured by the camera220. The display image 350 may be displayed on the display 210temporarily as a preview image to indicate to the user of the mobiledevice 205 how the scene that the camera 220 is receiving light from isframed in the field of view of the camera 220. The mobile device 205displays the circular indicator 510 overlaid over the display image 350.The circular indicator 510 outlines an area positioned over the camera220, and is not filled in.

It may be useful, in some software applications and associated userinterfaces running on the mobile device 205, to select certain parts ofan image. An image capture software application, and its associated userinterface, may be configured to allow selection of a region of a displayimage 350. For example, an image capture software application may beconfigured to select a selectable area 540 of the display image 350 inresponse to a touch input or a touch gesture input to a touchscreenwithin or around the selectable area 540 of the display image 350. Insome cases, the image capture software application may be configured toselect the selectable area 540 of the display image 350 in response to ahover input, a pointing input, or a hover gesture input in which a userhovers over (along the Z axis) and/or gestures over (along the Z axis)the selectable area 540. In some cases, the image capture softwareapplication may be configured to select the selectable area 540 of thedisplay image 350 in response to a voice input reciting a predefinedphrase recorded by a microphone. For example, the user of the mobiledevice 205 may speak the phrase “select the area over the camera” or asimilar phrase, which may act as a voice input. A microphone of themobile device 205 (or coupled to the mobile device 205) may record theuser saying this phrase as the voice input, and the mobile device 205(or a remote voice-to-text interpretation server) may recognize that thevoice input is saying a predefined phrase that triggers selection of theselectable area 540, triggering the mobile device 205 to select theselectable area 540.

In some examples, selection of the region of the preview image causesthe mobile device 205 to instruct the camera 220 to perform an autofocusoperation to adjust the focus of the camera 220 to improve focus on aportion of the photographed scene depicted in the selected selectablearea 540 of the display image 350. This provides an intuitive interfaceallowing a user of the mobile device to modify a focus of the camera,for instance, from being focused on an object in the background of thescene to being focused on an object in the foreground of the scene, orvice versa. For instance, in the display image 350 illustrated beingdisplayed on the display 210 in FIG. 4 , the selectable area 540 depictsa portion of a person’s face. The user may touch the touch-sensitivetouchscreen surface of the display 210 at the selectable area 540 of thedisplay 210, or hover over the surface of the display 210 at theselectable area 540 of the display 210, point at the selectable area 540of the display 210, gesture toward and/or around the selectable area 540of the display 210, or otherwise provide an input at the selectable area540 of the display 210. This input may be referred to as a touch input,a hover input, a pointing input, a gesture input, or another type ofinput based on the type of interaction with the display 210.The mobiledevice 205, detecting this input, identifies that the coordinates of theinput match coordinates of the display 210 upon which the region of thepreview image that depicts the person’s face is displayed. The mobiledevice 205 instructs the camera 220 to adjust the focus of the camera220 to improve focus on the person’s face, for instance by shifting froma first focus setting that optimizes focus on another object in thescene to a second focus setting that optimizes focus on the person’sface. The camera 220 can then capture a second image while the focus ofthe camera 220 is set to the second focus setting. Because of the changein focus, the second image may be more focused on the person’s face thanthe display image 350. The person’s face may appear clearer and sharperin the second image than in the display image 350.

In some examples, selection of an area or region of the display image350, such as the selectable area 540, can cause the mobile device 205 toinstruct the camera 220 to set one or more settings associated with oneor more properties of the camera 220 to improve application of the oneor more properties based on the portion of the photographed scenedepicted in the selected region or area of the display image 350. Forinstance one or more properties may include white balance, a black levelcompensation, a gamma correction, a gain, a color correction, a colorsaturation, a noise filter, a contrast control, a brightness control, atone mapping, a sharpness, a blurring, a red eye correction, or somecombination thereof. The camera 220 can capture a second image while theone or more settings associated with the one or more properties of thecamera 220 are set. The camera 220 can then capture a second image whilethe one or more settings associated with the one or more properties ofthe camera 220 are set. Assuming the selected region or area is theselectable area 540, because the one or more settings are set based onthe selection of the selectable area 540, the second image may usesettings that improve depiction of the person’s face compared to thedepiction of the person’s face in the display image 350. The person’sface may appear clearer and sharper in the second image than in thedisplay image 350.

Because the camera 220 is underneath the display 210 along the Z axis,receipt of certain inputs, such as inputs for selection of certainregions of the display image 350, occlude the view of the camera 220.Such inputs may include touch inputs, touch gesture inputs, hoverinputs, pointing inputs, hover gesture inputs, or any other kinds ofinputs discussed herein. For instance, a user providing a touch input ora touch gesture input by interacting with the touch-sensitivetouchscreen surface of the display 210 while the under-display camera220 is active may at least partially occlude the view of the camera 220with the user’s finger, stylus, hand, arm, or pointer by touching aportion of the display 210 that is positioned over the under-displaycamera 220. Even if the touch input is received at a particular area ofthe display 210 that is not directly over the camera 220, a portion ofthe user’s arm or hand may reach over the camera 220 and thus at leastpartially occlude the view of the camera 220 in order to reach theparticular area of the display 210. Similarly, a user may provide ahover input, a pointing input, or a hover gesture input by hovering overthe display 210, pointing at the display 210, and/or gesturing over thedisplay 210 with the user’s hand and/or a pointing device. Such hoverinputs, pointing inputs, and hover gesture inputs can likewise occludethe camera 220 by blocking light from reaching an area of the displaythrough which light would otherwise pass before reaching the camera 220.

In some examples, occlusion of the view of the camera 220 may refer tothe camera 220 being steered away from a target field of view (FOV). Themobile device 205 may determine the target FOV based on the softwareapplication being run by the mobile device 205. For example, if themobile device 205 is running a videoconferencing software application,the mobile device 205 may determine that the target FOV is any FOV ofthe camera 220 that includes a depiction of a face of a user. If themobile device 205 is moved (e.g., is accidentally dropped) such that theFOV of the camera 220 no longer fits the predetermined criteria of thetarget FOV in that the FOV of the camera 220 no longer includes adepiction of a face of a user. In some cases, the mobile device 205 maytreat any time during which the FOV of the camera 220 does not match thetarget FOV as a time during which the camera 220 is occluded.

The mobile device 205 of FIG. 5 includes an area selection hardwarebutton 530. The mobile device 205 may receive an area selection inputwhen the area selection hardware button 530 is pressed. In some cases,the area selection hardware button 530 can be replaced and/orsupplemented with another hardware user interface that receives theimage capture input. In response to receipt of the area selection input,the mobile device 205 may select a selectable area 540 of the displayimage 350 displayed on the display 210.

The mobile device 205 also includes an area selection user interface(UI) element 520 that is displayed on a UI area of the display 210,overlaid over the display image 350. The mobile device 205 may receivean area selection input at the area selection user UI element 520 of thedisplay 210. The area selection input may include a touch input, a touchgesture input, a hover input, a hover pointing input, a hover gestureinput, or any other type of input discussed herein. For example, a touchinput or a touch gesture input may include touch of the touch-sensitivetouchscreen surface of the display 210 receives a touch area at the areaselection user UI element 520 of the display 210. A hover input mayinclude a user hover over (in the Z axis) the area selection user UIelement 520 of the display 210. In some cases, a voice input may beused. In some cases, the image capture software application may beconfigured to select the selectable area 540 of the display image 350 inresponse to a voice input reciting a predefined phrase recorded by amicrophone as discussed herein.

As discussed above, selection of the selectable area 540 of the displayimage 350 using the area selection input from the area selectionhardware button 530 and/or the area selection UI element 520 may causethe mobile device 205 to instruct the camera 220 to set one or moresettings associated with one or more properties of the camera 220 toimprove application of the one or more properties based on the portionof the photographed scene depicted in the selectable area 540 of thedisplay image 350. For instance, selection of the selectable area 540 ofthe display image 350 using the area selection input may cause themobile device 205 to instruct the camera 220 to adjust its focus toimprove focus on a portion of the photographed scene depicted in theselectable area 540 of the display image 350. Because the user mayselect the selectable area 540 of the display image 350 by pressing thearea selection hardware button 530 and/or by providing an input at thearea selection UI element 520, the user no longer needs to occlude thecamera 220 to select the selectable area 540. While the selectable area540 illustrated in FIG. 5 is an area includes a portion of the display210 positioned directly over the camera 220, the selectable area neednot include the portion of the display 210 positioned directly over thecamera 220. In some cases, the selectable area 540 of the display image350 may include a portion of the display image 350 that a user may needto reach over the camera 220 to touch, hover over, point to, gesturetoward, gesture around, or otherwise provide an input to.

To make it clear that receipt of an input at the area selection UIelement 520 selects the selectable area 540 of the display image 350,the example area selection UI element 520 of FIG. 5 may include the word“select” over a visual copy of the selectable area 540 of the image.Alternatively, an input indicator (e.g., an arrow indicator, ahand-shaped indicator, a palm-shaped indicator, and the like) may beprovided to direct the user to touch the selectable area. The visualcopy of the selectable area 540 of the display image 350 may be modified(e.g., downscaled, upscaled, scaled, cropped, stretched, squished,skewed, distorted, or a combination thereof) to fit the size of the areaselection UI element 520. In some examples, the area selection UIelement 520 may act as a virtual button, and may select the entirety ofthe selectable area 540 when any part of the area selection UI element520 receives an input, such as a touch, a hover, a point, a gesture, anyinput type discussed herein, or a combination thereof. In some examples,the visual copy of the selectable area 540 of the display image 350 inthe area selection UI element 520 may act as an offset version of theselectable area 540. A user providing an input to a specific portion ofthe visual copy of the selectable area 540 of the display image 350 inthe area selection UI element 520 may select that specific portion ofthe selectable area 540 of the display image 350. In some examples, thearea selection UI element 520 may control a cursor. The user may swipe,slide, or otherwise move a touch input at the area selection UI element520 to move the cursor. The user may similarly use a swiping or slidinggesture while hovering over the area selection UI element 520 to movethe cursor. Selection of the selectable area 540 may be performed bymoving the cursor to the selectable area 540 of the display image 350and tapping the area selection UI element 520, pressing a button such asthe area selection hardware button 530, using a pointing movementgesture, or triggering another hardware interface element. In someexamples, the mobile device 205 may also allow a user to select theselectable area 540 of the display image 350 by drawing a shape aroundthe selectable area 540 using the cursor. In some examples, the mobiledevice 205 may also allow a user to select the selectable area 540 ofthe display image 350 by drawing a shape around the selectable area 540using touch input, hover input, gesture input, or another form of inputaround the selectable area 540.

The mobile device 205 of FIG. 5 also includes an image capture UIelement 320 and an image capture hardware button 330. These elements mayfunction similarly to the image capture UI element 320 and the imagecapture hardware button 330 of FIG. 3A, FIG. 3B, FIG. 3C, and FIG. 4 .While the image capture hardware button 330 and the area selectionhardware button 530 are illustrated along a single side of the mobiledevice 205, one or both can be located along different portions of themobile device 205. For example, one or both can be located along a rear,top, or bottom of the mobile device 205. Any other hardware interfaceelements used in place of the image capture hardware button 330 and/orthe area selection hardware button 530 may similarly be located along arear, top, or bottom of the mobile device 205.

FIG. 6 is a front view of a mobile device 205 that compensates forocclusion of an under-display camera 220 by a user’s hand 605 during atouch, hover, or gesture input. The legend 290 of FIG. 2A is alsoincluded in FIG. 6 . Like the mobile device 205 of FIG. 3A, FIG. 3B,FIG. 3C, FIG. 4 , and FIG. 5 , the mobile device 205 of FIG. 6 isillustrated displaying the display image 350 using its display 210. Thedisplay image 350 may be based on an image captured by the camera 220.The display image 350 may be displayed on the display 210 temporarily asa preview image to indicate to the user of the mobile device 205 how thescene that the camera 220 is receiving light from is framed in the viewof the camera 220.

A hand 605 is illustrated occluding the under-display camera 220 of themobile device 205 in FIG. 6 . The hand 605 has a finger outstretched,with the tip of the finger positioned over the camera 220, occludinglight from the scene from reaching the camera 220. The outstretchedfinger of the hand 605 may be touching a touch input area of thetouch-sensitive touchscreen surface of the display 210, hovering over aninput area of the display 210, pointing at an input area of the display210, gesturing toward an input area of the display, gesturing around aninput area of the display 210 or some combination thereof. The inputarea of the display 210 may be the selectable area 540 of FIG. 5 , oranother selectable area of the display image 350. This input to theinput area of the display may be used to select a selectable area of thedisplay image 350, such as the selectable area 540 of the display image350. The input area includes an area of the display 210 that ispositioned over the camera 220 through which incident light that reachesthe camera 220 passes. The input area may be positioned over the camera220, adjacent to the camera 220, and/or within a predetermined radius ofthe camera 220.

A timeline 620 of image frames captured by the camera 220 is illustratedin FIG. 6 . The timeline 620 of image frames captured by the camera 220includes a first image frame 610A, a second image frame 610B, a thirdimage frame 610C, a fourth image frame 610D, and a fifth image frame610E. In some examples, the image frames 610A-E in the timeline 620 maybe image frames that are captured consecutively by the camera 220. Insome examples, one or more additional image frames may exist between anytwo of the image frames in the timeline 620. The image frames 610A-E inthe timeline 620 depict a scene that is similar to the scene depicted inthe display image 350, but in some cases includes occlusions and/orother variations.

In the timeline 620, the first image frame 610A depicts a scene with aperson in the foreground in front of a background with buildings, grass,and trees. The first image frame 610A is not occluded. The second imageframe 610B is captured by the camera 220 after the first image frame610A. The second image frame 610B is partially occluded by an occlusion630A, which appears as a black or dark area covering a portion of thesecond image frame 610B. The occlusion 630A may, for example, be causedby a portion of the extended finger of the hand 605 approaching the viewof the camera 220 as the hand 605 is moving. The hand 605 blocks some ofthe light from the scene that would otherwise pass through the display210 and reach the camera 220. The third image frame 610C is captured bythe camera 220 after the second image frame 610B. The third image frame610C is fully occluded by an occlusion 630B, and is thus entirelycovered in blackness or darkness. The occlusion 630B may, for example,be caused by the extended finger of the hand 605 entirely occluding theview of the camera 220 as the hand 605 is moving. The hand 605 blocksmost or all of the light from the scene that would otherwise passthrough the display 210 and reach the camera 220. The fourth image frame610D is captured by the camera 220 after the third image frame 610C. Thefourth image frame 610D is partially occluded by an occlusion 630C,which appears as a black or dark area covering a portion of the fourthimage frame 610D. The hand 605 blocks some of the light from the scenethat would otherwise pass through the display 210 and reach the camera220. The occlusion 630C may, for example, be caused by a portion of theextended finger of the hand 605 leaving the view of the camera 220 asthe hand 605 is moving. The fifth image frame 610E is captured by thecamera 220 after the fourth image frame 610D. The fifth image frame 610Edepicts a similar scene as the first image frame 610A, though some minorchanges have occurred in the scene due to passage of time betweencapture of the first image frame 610A and the fifth image frame 610E.For instance, the scene depicted in the fifth image frame 610E includesa bird in the sky that is not present in the first image frame 610A.

Displaying the second image frame 610B, the third image frame 610C,and/or the fourth image frame 610D on the display 210 as preview imageswould result in display of the occlusions 630A-C on the on the display210. In some cases, a user may find the presence of the occlusions630A-C in a preview image unhelpful. In some examples, for the durationof a time in which the camera 220 is at least partially occluded, themobile device 205 displays the last unoccluded image frame from beforethe occlusion. In the timeline 620, the last unoccluded image frame frombefore the occlusion is the first image frame 610A. This is indicated bythe arrow 640. In some examples, for the duration of a time in which thecamera 220 is fully occluded (as in the occlusion 630B of the thirdimage frame 610C), the mobile device 205 displays the last partiallyunoccluded image frame from before the full occlusion 630B. In thetimeline 620, the last partially unoccluded image frame from before thefull occlusion 630B is the second image frame 610B. In some examples,the mobile device 205 displays the last image frame in which less than apredetermined maximum threshold percentage of the view of the camera 220is occluded for a duration of time while more than the predeterminedmaximum threshold percentage of the view of the camera 220 is occluded.

In some cases, a touch input or other input may be received while thecamera 220 is at least partially occluded. In some examples, mobiledevice 205 may receive a touch input in response to the extended fingerof the hand 605 touching a touch input area of the touch-sensitivetouchscreen surface of the display 210 at a time after capture of thefirst image frame 610A and before capture of the fifth image frame 610E.The mobile device 205 may select a selectable area of the image, such asthe selectable area 540 of FIG. 5 , based on the location of the display210 at which the touch input is received at least partially overlappingwith the selectable area 540. The mobile device 205 may instruct thecamera 220 to set one or more settings for one or more properties of thecamera 220 to adjust application of the one or more properties based onthe portion of the photographed scene depicted in the selected area inthe next unoccluded image frame after the occlusion is removed. The nextunoccluded image frame after the occlusion, in the timeline 620, is thefifth image frame 610E. Thus, the mobile device 205 may instruct thecamera 220 to set the one or more settings for the one or moreproperties of the camera 220 to adjust application of the one or moreproperties based on the portion of the photographed scene depicted inthe selected area in the last unoccluded image frame from before theocclusion.

In one illustrative example, while the camera 220 is at least partiallyoccluded, the mobile device 205 may still display the first image frame610A, and the user may tap a selectable area of the first image frame610A depicting the user’s face with the user’s hand 605. The mobiledevice 205 receives a touch input at the touch-sensitive touchscreensurface of the display 210 at a portion of the display that displays theselectable area of the first image frame 610A. The mobile device 205,detecting this touch input, identifies that the coordinates of the touchinput match the coordinates of the display 210 upon which the region ofthe first image frame 610A that depicts the user’s face is displayed.The mobile device 205 instructs the camera 220 to adjust the focus ofthe camera 220 to improve focus on the user’s face for the nextunoccluded image frame after the occlusion - the fifth image frame 610E.The mobile device 205 shifting from a first focus setting that optimizesfocus on another object in the scene to a second focus setting thatoptimizes focus on the user’s face. Once the camera 220 captures thefifth image frame 610E, the focus of the camera 220 may already be setto the second focus setting that optimizes focus on the user’s face.

FIG. 7 is a front view of a mobile device 205 with multipleunder-display cameras 220A-220B and under-display sensors 260A-260B inwhich the display 210 displays an interface area 770 for controlling themultiple under-display cameras 220A-220B and under-display sensors260A-260B. The legend 290 of FIG. 2A is also included in FIG. 7 . Likethe mobile device 205 of FIG. 2D, the mobile device 205 of FIG. 7includes a first under-display camera 220A and a second under-displaycamera 220B. The first under-display camera 220A and the secondunder-display camera 220B may each be any type of camera, and have anyof the camera properties, discussed herein with respect to the camera220, the first under-display camera 220A of FIG. 2D, the secondunder-display camera 220B of FIG. 2D, the image capture device 105A, theimage processing device 105B, the image capture and processing system100, any other camera discussed herein, or any combination thereof. Thefirst under-display camera 220A may also be referred to as the firstcamera 220A. The second under-display camera 220B may also be referredto as the second camera 220B.

Like the mobile device 205 of FIG. 3A, FIG. 3B, FIG. 3C, FIG. 4 , FIG. 5, and FIG. 6 , the mobile device 205 of FIG. 7 is illustrated displayingthe display image 350 using its display 210. The display image 350 maybe based on an image captured by the first camera 220A or by the secondcamera 220B. The display image 350 may be displayed on the display 210temporarily as a preview image to indicate to the user of the mobiledevice 205 the field of view of the first camera 220A or the field ofview of the second camera 220B.

The second camera 220B of FIG. 7 appears in the same position as thecamera 220 of FIG. 5 . The display 210 displays a circular indicator510B around the second camera 220B and within the selectable area 540Bover the second camera 220B. The second indicator 510B of FIG. 7 appearscircular and the same as the indicator 510 of FIG. 5 . The secondselectable area 540B over the second camera 220B of FIG. 7 is alsocircular, and appears to have the same shape and size and position asthe selectable area 540 over the camera 220 of FIG. 5 . In otherexamples, the second camera 220B, the second indicator 510B, and/or thesecond selectable area 540B may be different than those in FIG. 5 .

The first camera 220A of FIG. 7 appears near the top of the display 210,similarly to the first camera 220A in FIG. 2D, the camera 220 in FIG.3A, the camera 220 in FIG. 3B, the camera 220 in FIG. 3C, and the camera220 in FIG. 4 . The display 210 displays a first indicator 510A aroundthe first camera 220A that is a rounded rectangle and lies within afirst selectable area 540A over the first camera 220A. The firstselectable area 540A over the first camera 220A of FIG. 7 is a largerounded rectangle that includes a majority of the sky as depicted in thedisplay image 350. In other examples, the second camera 220B, the secondindicator 510B, and/or the second selectable area 540B may be differentthan those in FIG. 5 .

The interface area 770 appears as a shaded translucent area overlaidover the image displayed on the display 210. The interface area 770includes the image capture UI element 320, an first area selection UIelement 720A, a second area selection UI element 720B, and a sensorcontrol UI element 740. The first area selection UI element 720A and thesecond area selection UI element 720B of FIG. 7 function similarly tothe area selection UI element 520 of FIG. 5 . Likewise, the first areaselection hardware button 730A and the second area selection hardwarebutton 730B of FIG. 7 function similarly to the area selection hardwarebutton 530 of FIG. 5 . The mobile device 205 selects the firstselectable area 540A upon receipt of an input at the first areaselection UI element 720A. The mobile device 205 selects the firstselectable area 540A upon receipt of an input at the first areaselection hardware button 730A. A user wishing to select the sky asdepicted in the display image 350 can provide an input to the first areaselection UI element 720A and/or can provide an input to the first areaselection hardware button 730A to do so. The mobile device 205 selectsthe second selectable area 540B upon receipt of an input at the secondarea selection UI element 720B. The mobile device 205 selects the secondselectable area 540B upon receipt of an input at the second areaselection hardware button 730B. A user wishing to select the person’sface as depicted in the display image 350 can provide an input to thesecond area selection UI element 720B and/or can provide an input to thesecond area selection hardware button 730B to do so.

In some cases, one of the first camera 220A or the second camera 220Bcan be disabled for a certain purpose or software application. In somecases, one of the first camera 220A or the second camera 220B can beactivated for a certain purpose or software application. For example, ifthe mobile device 205 is running a videoconferencing software that isconducting a videoconference, the first camera 220A may be disabled foruse in the videoconference, and the second camera 220B may be activatedfor use in the videoconference. In a videoconference, the user of themobile device 205 is videoconferencing with a second user. The user ofthe mobile device 205 generally looks at the second user’s face, whichis displayed on the display 210. The second user’s face may be displayedapproximately centered on the display 210. During such avideoconference, the user of the mobile device 205 may naturally looktoward the center of the display 210 because the user is looking at theother user’s face. Because the second camera 220B is closer to thecenter of the display 210 than the first camera 220A, a user that islooking at the other user’s face as displayed in the display 210 is alsolooking either directly at the second camera 220B or at a point that iscloser to the second camera 220B than to the first camera 220A. An imagecaptured by the second camera 220B while the user is looking at theuser’s face as displayed in the display 210 therefore depicts the usermaking eye contact with the second camera 220B, or making eye contactwith a point close to the second camera 220B. By using the second camera220B rather than the first camera 220A in the videoconference, the imageof the user making eye contact captured by the second camera 220B issent to the second user’s device, and is displayed at the second user’sdevice. The image captured by the second camera 220B, when displayed onthe second user’s device, then appears to be making eye contact with thesecond user as the second user views the image as displayed on thesecond user’s device. In videoconferencing, there is generally no needto touch, hover over, or otherwise interact with the center of thedisplay 210, so occlusion of the second camera 220B is unlikely duringvideoconferencing. Thus, use of the second camera 220B forvideoconferencing improves videoconferencing by improving the appearanceof eye contact between the user and the second user. This appearance ofeye contact generally cannot be achieved with cameras that are moreoffset, such as the first camera 220A, other than through imagemanipulation that modifies depictions of the user’s eyes. Such imagemanipulation can appear unnatural and produce an undesirable uncannyvalley effect.

On the other hand, a software application in which user selection ofareas of the display image 350 are anticipated and/or expected may usethe first camera 220A and disable the second camera 220B. The firstcamera 220A which is closer than the second camera 220B to the top sideof the mobile device 205. Thus, the first camera 220A is less likelythan the second camera 220B to be occluded by a user holding the mobiledevice 205 with the bottom side of the mobile device 205 being closer tothe user’s body than the top side of the mobile device 205. Forinstance, a software application in which a user draws on or otherwiseedits their own selfie may user the first camera 220A and disable thesecond camera 220B, as inputs to select areas of a display image 350 areanticipated and/or expected in such a software application.

In some cases, the mobile device 205 may switch which of the cameras isactive and which of the cameras is disabled based on occlusion of thecameras. For instance, if the first camera 220A is active, and themobile device 205 detects an occlusion similar to one of the occlusions630A-C occluding light from being received by the first camera 220A, themobile device 205 may disable the first camera 220A and activate thesecond camera 220B. Likewise, if the second camera 220B is active, andthe mobile device 205 detects an occlusion similar to one of theocclusions 630A-C occluding light from being received by the secondcamera 220B, the mobile device 205 may disable the second camera 220Band activate the first camera 220A.

Similarly to the under-display sensor 260 of the mobile device 205 ofFIG. 2D, the mobile device 205 of FIG. 7 includes a first under-displaysensor 260A and a second under-display sensor 260B. The firstunder-display sensor 260A and the second under-display sensor 260B mayeach be any type of sensor, and have any of the sensor properties,discussed herein with respect to the under-display sensor 260, any othercamera or sensor discussed herein, or any combination thereof. Themobile device 205 can trigger use and/or control over one or both of thesensor 260A and/or the sensor 260B upon receipt of an input at thesensor control UI element 740, and/or upon receipt of an input at thesensor control hardware button 775. While a single sensor control UIelement 740 and a single sensor control hardware button 775 areillustrated in FIG. 7 , in some cases a mobile device 205 may include aseparate sensor control UI element 740 for each of the sensors 260A-260Band/or a separate sensor control hardware button 775 for each of thesensors 260A-260B. The mobile device 205 can select a selectable area ofthe display image 350 over the first sensor 260A upon receipt of aninput at a sensor control UI element 740 and/or a sensor controlhardware button 775 corresponding to the first sensor 260A. The mobiledevice 205 can select a selectable of the display image 350 over thesecond sensor 260B upon receipt of an input at a sensor control UIelement 740 and/or a sensor control hardware button 775 corresponding tothe second sensor 260B.

The mobile device 205 may activate the first sensor 260A and the secondsensor 260B simultaneously, or within the same time window, so that themobile device 205 can gather more sensor data and obtain a more completeand accurate understanding of the environment. For example, if the firstsensor 260A is a RADAR sensor and the second sensor 260B is a LIDARsensor, then the mobile device 205 can activate both sensors 260A-260Band can use sensor measurement data from both sensors 260A-260B and/orfrom the cameras 220A-220B to obtain detailed and accurate depthinformation of the scene depicted in the display image 350. Differentones of the sensors 260A-260B can be activated and disabled at differenttimes, for instance based on what software application is running on themobile device 205. One software application may trigger the mobiledevice 205 to activate the first sensor 260A but disable the secondsensor 260B. Another software application may trigger the mobile device205 to activate the second sensor 260B but disable the first sensor260A. Another software application may trigger the mobile device 205 toactivate both the first sensor 260A and the second sensor 260B. Anothersoftware application may trigger the mobile device 205 to disable boththe first sensor 260A and the second sensor 260B. The mobile device 205can in some situations detect an occlusion of one of the sensors260A-260B, in which the mobile device 205 can disable the occludedsensor and activate the other (non-occluded) sensor. For example, themobile device 205 can detect an occlusion of one of the sensors260A-260B when a sensor returns a sensor measurement indicatingdetection of an object that is less than a threshold distance away fromthe sensor. In some cases, a sensor of the sensors 260A-260B may bepositioned within a threshold distance of a camera of the cameras220A-220B. If the mobile device 205 detects an occlusion of such acamera (e.g., as in the occlusion 630A-C of FIG. 6 ), the mobile device205 can assume that the sensor that is positioned within the thresholddistance of the camera. For example, in FIG. 7 , the sensors 260A-260Bmay be within the threshold distance of the first camera 220A. If themobile device 205 detects an occlusion of the first camera 220A, themobile device 205 may assume that the sensors 260A-260B are alsooccluded. This type of occlusion detection may be useful when one ormore of the sensors 260A-260B are of a sensor type that may be difficultto detect occlusion of, such as microphones.

FIG. 8 is a flow diagram illustrating an image processing technique 800.The image processing technique 800 illustrated by the flow diagram ofFIG. 8 may be performed by a device. The device may be a mobile device205, an image capture and processing system 100, an image capture device105A, an image processing device 105B, one or more network servers of acloud service, a computing system 1000, or some combination thereof.

At operation 805, the device receives an image captured by a camera 220.The camera 220 captures the image based on incident light 245 receivedby the camera 220. The camera 220 is positioned relative to a display210 to receive the incident light 245 that passes through a portion ofthe display 210 before reaching the camera 220. The portion of thedisplay 210 may be, for example, a light-passing area of the display 210as in the light-passing area 285. Reaching the camera can includeimpinging an image sensor 130 of the camera 220. The camera can bepositioned adjacent to the display along an axis that is perpendicularto a planar surface of the display, such as the Z-axis of the legend 290of FIG. 2A in the context of the mobile device 205 of FIG. 2A.

The display 210 can be a touchscreen display. The device can use atouch-sensitive surface of the touchscreen display to detect a touchinput or a touch gesture input received at the touch-sensitive surfaceof the touchscreen display. The display 210 can include one or moresensors 260. The one or more sensors 260 can include one or more activedepth sensors. The device can use the one or more active depth sensorsto detect a hover input, a hover pointing input, or a hover gestureinput over the display 210.

At operation 810, the device displays, on the display 210, a displayimage 350 that is based on the image. In some cases, the devicegenerates the display image 350 based on the image that is captured bythe camera 220 before displaying the display image 350 on the display220. Generating the display image 350 may include modifying a size ofthe image using at least one of scaling and/or cropping. Scaling caninclude upscaling, downscaling, and/or stretching. Modifying the size ofthe image may include modifying the size of the image such that thedisplay image 350 is displayed on the display 210 other than on anindicator area. Generating the display image 350 may also includeprocessing the image, for instance by modifying at least one of abrightness, a contrast, a color saturation, a tone, a white balance, ablack balance, a level, a gamma correction, a gain, a color correction,a noise filter, a sharpness, a blurring, and a red eye correction.

At operation 815, the device displays an indicator overlaid over thedisplay image 350 at an indicator area of the display 210 while thedisplay 210 is displaying the display image 350. The indicator areaincludes at least a subset of the portion of the display. Examples ofthe indicator of operation 815 include the indicator 310A, the indicator310B, the indicator 310C, the indicator arrows 340, the indicator 410,and the indicator 510, and/or combinations thereof. The indicator areaof the display can be at least as large as a lens of the camera. Theindicator can be at least as large as a lens of the camera.

In some cases, the device may display the indicator overlaid over thedisplay image 350 at the indicator area automatically. In some cases,the device receives an input, and the device displays the indicatoroverlaid over the display image 350 at the indicator area in response toreceipt of the input. The input may include at least one of a touch of atouch-sensitive touchscreen surface of the display 210, a predefinedgesture across the touch-sensitive touchscreen surface of the display210, a hover over the display 210, a predefined gesture over the display210, a voice input reciting a predefined phrase recorded by amicrophone, or a combination thereof.

The indicator can include a shape around at least part of portion of thedisplay 210. Displaying the indicator can include causing a plurality ofpixels in the indicator area to display a color. The color can be black,white, grey, red, green, blue, yellow, or some mixture of these colors.Displaying the indicator can include causing a plurality of pixels inthe indicator area to display a pattern or secondary image distinct fromthe image and from the display image 350. The indicator can be animated.For example, the indicator can blink or pulsate.

Displaying the indicator overlaid over the display image 350 at theindicator area while the display 210 is displaying the display image 350can include modifying the display image 360 to merge the indicator withthe display image 350.

The device can display an image capture user interface (UI) element 320over an image capture UI area on the display 210. The device can receivean input at the image capture UI area. The input can be one of a touchinput, a hover input, a gesture input, or any other input type discussedherein. The device can receive a second image captured by the camera 220in response to receiving the input at the image capture UI area. Thedevice can store the second image to a non-transitory computer-readablestorage medium.

The device can receive an input at an area selection user interface (UI)area. The input can be at least one of a touch input, a hover input, agesture input, or any other input type discussed herein. The device canselect a selection area of the display image 350 in response toreceiving the input at the area selection UI area. The selectable areas540, 540A, and 540B are examples of the selection area. The areaselection UI area is distinct from the selection area of the displayimage. The selection area of the display image can include at least aportion of the indicator area. In some cases, the device displays anarea selection UI element 320 at the area selection UI area. In somecases, the device generates the area selection UI element 320 to depicta copy of at least a portion of the selection area.

The device may set an image capture parameter to a particular settingthat is determined based on the selection area of the display image 350in response to selection of the selection area. The image captureparameter can includes at least one of a focus or an exposure parameter.The exposure parameter may include at least one of an exposure time, ananalog gain, a digital gain, an aperture size, an ISO, or a combinationthereof. The device may set an image processing property to a particularsetting that is determined based on the selection area of the displayimage in response to selection of the selection area. The imageprocessing property may include at least one of a brightness, acontrast, a color saturation, a tone, a white balance, a black balance,a black level compensation, a level, a gamma correction, a gain, a colorcorrection, a noise filter, a sharpness, a blurring, a red eyecorrection, or a combination thereof.

The device can receive a second image captured by the camera 220 aftercapture of the image. The device can determine that the second image isoccluded by an occlusion, such as the occlusion 630A-C. The device candisplay the display image 350 on the display 210 in response todetermining that the second image is occluded by the occlusion. Forexample, because the second image is occluded by the occlusion, thedevice can choose to display the display image 350 on the display 210again instead of displaying the second image or a second display imagethat is based on the second image. In some cases, the device can receivean input, where the occlusion is associated with receipt of the input.The input can be at least one of a touch input, a hover input, a gestureinput, or any other input type discussed herein. The device can select aselection area of the display image based on the input. The device candetermine a setting based on the selection area of the display image,and can apply the setting. The device can receive a third image capturedby the camera 220 while the setting is applied and after capture of thesecond image.

In some cases, the device can receive multiple cameras. In someexamples, the device can receive a second image captured by a secondcamera. The second camera can capture the second image based onsecondary incident light received by the second camera. The secondcamera can be positioned relative to the display 210 to receive thesecondary incident light that passes through a second portion of thedisplay 210 (e.g., a second light-passing area of the display 210)before reaching the second camera. Reaching the second camera caninclude impinging a second image sensor of the second camera. In someexamples, the device can determine that the second image is occluded byan occlusion, and can display the display image 350 on the display 210in response to determining that the second image is occluded by theocclusion. In some examples, the device can determine depth informationcorresponding to one or more objects depicted in the image and in thesecond image by processing the image and the second image, for exampleusing stereoscopic image processing.

FIG. 9 is a flow diagram illustrating an image processing technique 900.The image processing technique 900 illustrated by the flow diagram ofFIG. 9 may be performed by a device. The device may be a mobile device205, an image capture and processing system 100, an image capture device105A, an image processing device 105B, one or more network servers of acloud service, a computing system 1000, or some combination thereof.

At operation 905, the device receives an image captured by a camera 220.The camera 220 captures the image based on incident light 245 receivedby the camera 220. The camera 220 is positioned relative to atouchscreen display 210 to receive the incident light 245 that passesthrough a light-passing area 285 of the touchscreen display 210 beforereaching the camera 220. Reaching the camera can include impinging animage sensor 130 of the camera 220. The light-passing area 285 of thetouchscreen display 210 of operation 905 may be an example of theportion of the display of operation 805.

At operation 910, the device displays, on the touchscreen display 210, adisplay image 350 based on the image.

At operation 915, the device displays an area selection user interface(UI) element on the touchscreen display 210. The area selection UIelement of operation 915 may include, for example, the area selection UIelement 520 of FIG. 5 , area selection UI element 720A of FIG. 7 , orthe area selection UI element 720B of FIG. 7 .

At operation 920, the device receives a touch input to the touchscreendisplay 210 at an area selection input area that overlaps with the areaselection UI element. In some cases, the touch input may be supplementedwith, or replaced with, a hover input.

At operation 925, the device selects a selection area of the displayimage 350 in response to receiving the touch input. The selection areamay be a selectable area including at least part of the light-passingarea, such as the selectable area 540 of FIG. 5 , the selectable area540A of FIG. 7 , or the selectable area 540B of FIG. 7 .

At operation 930, the device sets a camera property of the camera 220 toa setting that is determined based on the selection area of the displayimage 350 in response to selection of the selection area. The cameraproperty of the camera 220 includes at least one of an image captureproperty or an image processing property. The image capture property mayinclude at least one of a focus and an exposure parameter. The exposureparameter may include at least one of an exposure time, an analog gain,a digital gain, an aperture size, an ISO, or a combination thereof. Theimage processing property may include at least one of a brightness, acontrast, a color saturation, a tone, a white balance, a black balance,a black level compensation, a level, a gamma correction, a gain, a colorcorrection, a noise filter, a sharpness, a blurring, a red eyecorrection, or a combination thereof.

In some cases, at least a subset of the image processing techniques 800and 900 illustrated by the flow diagrams of FIGS. 8-9 may be performedremotely by one or more network servers of a cloud service. In someexamples, the processes described herein (e.g., the image processingtechniques 800 and 900 and/or other processes described herein) may beperformed by a computing device or apparatus. In one example, the imageprocessing techniques 800 and 900 can be performed by the image capturedevice 105A of FIG. 1 . In another example, the image processingtechniques 800 and 900 can be performed by the image processing device105B of FIG. 1 . The image processing techniques 800 and 900 can also beperformed by the image capture and processing system 100 of FIG. 1 . Theimage processing techniques 800 and 900 can also be performed by themobile device 205 of any of FIGS. 2A, 2B, 2C, 3A, 3B, 3C, 4, 5, 6,and/or 7 . The image processing techniques 800 and 900 can be performedby a computing device with the architecture of the computing system 1000shown in FIG. 10 . The computing device can include any suitable device,such as a mobile device (e.g., a mobile phone), a desktop computingdevice, a tablet computing device, a wearable device (e.g., a VRheadset, an AR headset, AR glasses, a network-connected watch orsmartwatch, or other wearable device), a server computer, an autonomousvehicle or computing device of an autonomous vehicle, a robotic device,a television, and/or any other computing device with the resourcecapabilities to perform the processes described herein, including theimage processing techniques 800 and 900. In some cases, the computingdevice or apparatus may include various components, such as one or moreinput devices, one or more output devices, one or more processors, oneor more microprocessors, one or more microcomputers, one or morecameras, one or more sensors, and/or other component(s) that areconfigured to carry out the steps of processes described herein. In someexamples, the computing device may include a display, a networkinterface configured to communicate and/or receive the data, anycombination thereof, and/or other component(s). The network interfacemay be configured to communicate and/or receive Internet Protocol (IP)based data or other type of data.

The components of the computing device can be implemented in circuitry.For example, the components can include and/or can be implemented usingelectronic circuits or other electronic hardware, which can include oneor more programmable electronic circuits (e.g., microprocessors,graphics processing units (GPUs), digital signal processors (DSPs),central processing units (CPUs), and/or other suitable electroniccircuits), and/or can include and/or be implemented using computersoftware, firmware, or any combination thereof, to perform the variousoperations described herein.

The processes illustrated by conceptual diagrams and flow diagrams ofFIGS. 8-9 are organized as logical flow diagrams, the operation of whichrepresents a sequence of operations that can be implemented in hardware,computer instructions, or a combination thereof. In the context ofcomputer instructions, the operations represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by one or more processors, perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular data types. The order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the processes.

Additionally, the processes illustrated by conceptual diagrams and flowdiagrams of FIGS. 8-9 and/or other processes described herein may beperformed under the control of one or more computer systems configuredwith executable instructions and may be implemented as code (e.g.,executable instructions, one or more computer programs, or one or moreapplications) executing collectively on one or more processors, byhardware, or combinations thereof. As noted above, the code may bestored on a computer-readable or machine-readable storage medium, forexample, in the form of a computer program comprising a plurality ofinstructions executable by one or more processors. The computer-readableor machine-readable storage medium may be non-transitory.

While FIGS. 2A-2C, 3A-3C, 4, 5, 6, and 7 all illustrate a mobile device205, it should be understood that the mobile device 205 is just oneexample of a device that includes the technologies discussed hereinand/or performs the techniques discussed herein. The technologies andtechniques discussed herein, including the image processing techniques800 and 900, can be performed by a mobile device 205, an image captureand processing system 100, an image capture device 105A, an imageprocessing device 105B, one or more network servers of a cloud service,a computing system 1000, or some combination thereof.

FIG. 10 is a diagram illustrating an example of a system forimplementing certain aspects of the present technology. In particular,FIG. 10 illustrates an example of computing system 1000, which can befor example any computing device making up internal computing system, aremote computing system, a camera, or any component thereof in which thecomponents of the system are in communication with each other usingconnection 1005. Connection 1005 can be a physical connection using abus, or a direct connection into processor 1010, such as in a chipsetarchitecture. Connection 1005 can also be a virtual connection,networked connection, or logical connection.

In some embodiments, computing system 1000 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 1000 includes at least one processing unit (CPU orprocessor) 1010 and connection 1005 that couples various systemcomponents including system memory 1015, such as read-only memory (ROM)1020 and random access memory (RAM) 1025 to processor 1010. Computingsystem 1000 can include a cache 1012 of highspeed memory connecteddirectly with, in close proximity to, or integrated as part of processor1010.

Processor 1010 can include any general purpose processor and a hardwareservice or software service, such as services 1032, 1034, and 1036stored in storage device 1030, configured to control processor 1010 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. Processor 1010 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction, computing system 1000 includes an inputdevice 1045, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 1000 can also include output device 1035, which can be one ormore of a number of output mechanisms. In some instances, multimodalsystems can enable a user to provide multiple types of input/output tocommunicate with computing system 1000. Computing system 1000 caninclude communications interface 1040, which can generally govern andmanage the user input and system output. The communication interface mayperform or facilitate receipt and/or transmission wired or wirelesscommunications using wired and/or wireless transceivers, including thosemaking use of an audio jack/plug, a microphone jack/plug, a universalserial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernetport/plug, a fiber optic port/plug, a proprietary wired port/plug, aBLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE)wireless signal transfer, an IBEACON® wireless signal transfer, aradio-frequency identification (RFID) wireless signal transfer,near-field communications (NFC) wireless signal transfer, dedicatedshort range communication (DSRC) wireless signal transfer, 802.11 Wi-Fiwireless signal transfer, wireless local area network (WLAN) signaltransfer, Visible Light Communication (VLC), Worldwide Interoperabilityfor Microwave Access (WiMAX), Infrared (IR) communication wirelesssignal transfer, Public Switched Telephone Network (PSTN) signaltransfer, Integrated Services Digital Network (ISDN) signal transfer,3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hocnetwork signal transfer, radio wave signal transfer, microwave signaltransfer, infrared signal transfer, visible light signal transfer,ultraviolet light signal transfer, wireless signal transfer along theelectromagnetic spectrum, or some combination thereof. Thecommunications interface 1040 may also include one or more GlobalNavigation Satellite System (GNSS) receivers or transceivers that areused to determine a location of the computing system 1000 based onreceipt of one or more signals from one or more satellites associatedwith one or more GNSS systems. GNSS systems include, but are not limitedto, the US-based Global Positioning System (GPS), the Russia-basedGlobal Navigation Satellite System (GLONASS), the China-based BeiDouNavigation Satellite System (BDS), and the Europe-based Galileo GNSS.There is no restriction on operating on any particular hardwarearrangement, and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1030 can be a non-volatile and/or non-transitory and/orcomputer-readable memory device and can be a hard disk or other types ofcomputer readable media which can store data that are accessible by acomputer, such as magnetic cassettes, flash memory cards, solid statememory devices, digital versatile disks, cartridges, a floppy disk, aflexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, anyother magnetic storage medium, flash memory, memristor memory, any othersolid-state memory, a compact disc read only memory (CD-ROM) opticaldisc, a rewritable compact disc (CD) optical disc, digital video disk(DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographicoptical disk, another optical medium, a secure digital (SD) card, amicro secure digital (microSD) card, a Memory Stick® card, a smartcardchip, a EMV chip, a subscriber identity module (SIM) card, amini/micro/nano/pico SIM card, another integrated circuit (IC)chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM(DRAM), read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cachememory (L1/L2/L3/L4/L5/L#), resistive random-access memory (RRAM/ReRAM),phase change memory (PCM), spin transfer torque RAM (STT-RAM), anothermemory chip or cartridge, and/or a combination thereof.

The storage device 1030 can include software services, servers,services, etc., that when the code that defines such software isexecuted by the processor 1010, it causes the system to perform afunction. In some embodiments, a hardware service that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1010, connection 1005, output device 1035,etc., to carry out the function.

As used herein, the term “computer-readable medium” includes, but is notlimited to, portable or non-portable storage devices, optical storagedevices, and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A computer-readable medium mayinclude a non-transitory medium in which data can be stored and thatdoes not include carrier waves and/or transitory electronic signalspropagating wirelessly or over wired connections. Examples of anon-transitory medium may include, but are not limited to, a magneticdisk or tape, optical storage media such as compact disk (CD) or digitalversatile disk (DVD), flash memory, memory or memory devices. Acomputer-readable medium may have stored thereon code and/ormachine-executable instructions that may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.Information, arguments, parameters, data, etc. may be passed, forwarded,or transmitted using any suitable means including memory sharing,message passing, token passing, network transmission, or the like.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Specific details are provided in the description above to provide athorough understanding of the embodiments and examples provided herein.However, it will be understood by one of ordinary skill in the art thatthe embodiments may be practiced without these specific details. Forclarity of explanation, in some instances the present technology may bepresented as including individual functional blocks including functionalblocks comprising devices, device components, steps or routines in amethod embodied in software, or combinations of hardware and software.Additional components may be used other than those shown in the figuresand/or described herein. For example, circuits, systems, networks,processes, and other components may be shown as components in blockdiagram form in order not to obscure the embodiments in unnecessarydetail. In other instances, well-known circuits, processes, algorithms,structures, and techniques may be shown without unnecessary detail inorder to avoid obscuring the embodiments.

Individual embodiments may be described above as a process or methodwhich is depicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations can beperformed in parallel or concurrently. In addition, the order of theoperations may be re-arranged. A process is terminated when itsoperations are completed, but could have additional steps not includedin a figure. A process may correspond to a method, a function, aprocedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

Processes and methods according to the above-described examples can beimplemented using computer-executable instructions that are stored orotherwise available from computer-readable media. Such instructions caninclude, for example, instructions and data which cause or otherwiseconfigure a general purpose computer, special purpose computer, or aprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware,source code, etc. Examples of computer-readable media that may be usedto store instructions, information used, and/or information createdduring methods according to described examples include magnetic oroptical disks, flash memory, USB devices provided with non-volatilememory, networked storage devices, and so on.

Devices implementing processes and methods according to thesedisclosures can include hardware, software, firmware, middleware,microcode, hardware description languages, or any combination thereof,and can take any of a variety of form factors. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks (e.g., a computer-programproduct) may be stored in a computer-readable or machine-readablemedium. A processor(s) may perform the necessary tasks. Typical examplesof form factors include laptops, smart phones, mobile phones, tabletdevices or other small form factor personal computers, personal digitalassistants, rackmount devices, standalone devices, and so on.Functionality described herein also can be embodied in peripherals oradd-in cards. Such functionality can also be implemented on a circuitboard among different chips or different processes executing in a singledevice, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are example means for providing the functionsdescribed in the disclosure.

In the foregoing description, aspects of the application are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the application is not limited thereto. Thus,while illustrative embodiments of the application have been described indetail herein, it is to be understood that the inventive concepts may beotherwise variously embodied and employed, and that the appended claimsare intended to be construed to include such variations, except aslimited by the prior art. Various features and aspects of theabove-described application may be used individually or jointly.Further, embodiments can be utilized in any number of environments andapplications beyond those described herein without departing from thebroader spirit and scope of the specification. The specification anddrawings are, accordingly, to be regarded as illustrative rather thanrestrictive. For the purposes of illustration, methods were described ina particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed.

One of ordinary skill will appreciate that the less than (“<”) andgreater than (“>”) symbols or terminology used herein can be replacedwith less than or equal to (“≤”) and greater than or equal to (“≥”)symbols, respectively, without departing from the scope of thisdescription.

Where components are described as being “configured to” perform certainoperations, such configuration can be accomplished, for example, bydesigning electronic circuits or other hardware to perform theoperation, by programming programmable electronic circuits (e.g.,microprocessors, or other suitable electronic circuits) to perform theoperation, or any combination thereof.

The phrase “coupled to” refers to any component that is physicallyconnected to another component either directly or indirectly, and/or anycomponent that is in communication with another component (e.g.,connected to the other component over a wired or wireless connection,and/or other suitable communication interface) either directly orindirectly.

Claim language or other language reciting “at least one of” a set and/or“one or more” of a set indicates that one member of the set or multiplemembers of the set (in any combination) satisfy the claim. For example,claim language reciting “at least one of A and B” means A, B, or A andB. In another example, claim language reciting “at least one of A, B,and C” means A, B, C, or A and B, or A and C, or B and C, or A and B andC. The language “at least one of” a set and/or “one or more” of a setdoes not limit the set to the items listed in the set. For example,claim language reciting “at least one of A and B” can mean A, B, or Aand B, and can additionally include items not listed in the set of A andB.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software,firmware, or combinations thereof. To clearly illustrate thisinterchangeability of hardware and software, various illustrativecomponents, blocks, modules, circuits, and steps have been describedabove generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present application.

The techniques described herein may also be implemented in electronichardware, computer software, firmware, or any combination thereof. Suchtechniques may be implemented in any of a variety of devices such asgeneral purposes computers, wireless communication device handsets, orintegrated circuit devices having multiple uses including application inwireless communication device handsets and other devices. Any featuresdescribed as modules or components may be implemented together in anintegrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a computer-readable data storage mediumcomprising program code including instructions that, when executed,performs one or more of the methods described above. Thecomputer-readable data storage medium may form part of a computerprogram product, which may include packaging materials. Thecomputer-readable medium may comprise memory or data storage media, suchas random access memory (RAM) such as synchronous dynamic random accessmemory (SDRAM), read-only memory (ROM), non-volatile random accessmemory (NVRAM), electrically erasable programmable read-only memory(EEPROM), FLASH memory, magnetic or optical data storage media, and thelike. The techniques additionally, or alternatively, may be realized atleast in part by a computer-readable communication medium that carriesor communicates program code in the form of instructions or datastructures and that can be accessed, read, and/or executed by acomputer, such as propagated signals or waves.

The program code may be executed by a processor, which may include oneor more processors, such as one or more digital signal processors(DSPs), general purpose microprocessors, an application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), orother equivalent integrated or discrete logic circuitry. Such aprocessor may be configured to perform any of the techniques describedin this disclosure. A general purpose processor may be a microprocessor;but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Accordingly, the term “processor,” as used herein mayrefer to any of the foregoing structure, any combination of theforegoing structure, or any other structure or apparatus suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated software modules or hardware modules configured for encodingand decoding, or incorporated in a combined video encoder-decoder(CODEC).

Illustrative aspects of the disclosure include:

Aspect 1: An apparatus for image processing, the apparatus comprising: amemory; and one or more processors coupled to the memory and configuredto: receive an image captured by a camera based on incident lightreceived by the camera, wherein the camera is positioned relative to adisplay to receive the incident light that passes through a portion ofthe display before reaching the camera; display, on the display, adisplay image that is based on the image; and display an indicatoroverlaid over the display image at an indicator area of the displaywhile the display is displaying the display image, wherein the indicatorarea includes at least a subset of the portion of the display.

Aspect 2: The apparatus of Aspect 1, wherein the apparatus is at leastone of a mobile device, a wireless communication device, and a cameradevice.

Aspect 3: The apparatus of any of Aspects 1 to 2, wherein the apparatusincludes at least one of the camera and the display.

Aspect 4: The apparatus of any of Aspects 1 to 3, wherein at least oneof the indicator and the indicator area of the display are at least aslarge as an area of a lens of the camera.

Aspect 5: The apparatus of any of Aspects 1 to 4, wherein the one ormore processors are further configured to: receive an input, wherein theindicator is displayed over the indicator area in response to receipt ofthe input.

Aspect 6: The apparatus of Aspect 5, wherein the input includes at leastone of a touch of a touch-sensitive touchscreen surface of the display,a predefined gesture across the touch-sensitive touchscreen surface ofthe display, a hover over the display, a predefined gesture over thedisplay, a voice input reciting a predefined phrase recorded by amicrophone.

Aspect 7: The apparatus of any of Aspects 1 to 6, wherein the one ormore processors are further configured to: generate the display image bymodifying a size of the image using at least one of scaling andcropping.

Aspect 8: The apparatus of Aspect 7, wherein modifying the size of theimage includes modifying the size of the image such that the displayimage is displayed on the display other than on the indicator area.

Aspect 9: The apparatus of any of Aspects 1 to 8, wherein the indicatorincludes a shape around at least part of the portion of the display.

Aspect 10: The apparatus of any of Aspects 1 to 9, wherein the indicatoris animated.

Aspect 11: The apparatus of any of Aspects 1 to 10, wherein displayingthe indicator overlaid over the display image at the indicator areawhile the display is displaying the display image includes modifying thedisplay image to merge the indicator with the display image.

Aspect 12: The apparatus of any of Aspects 1 to 11, wherein the displayis a touchscreen.

Aspect 13: The apparatus of any of Aspects 1 to 12, wherein the one ormore processors are further configured to: display an image capture userinterface (UI) element over an image capture UI area on the display;receive an input at the image capture UI area, wherein the input is oneof a touch input, a hover input, and a gesture input; receive a secondimage captured by the camera in response to receiving the input at theimage capture UI area; and store the second image to a non-transitorycomputer-readable storage medium.

Aspect 14: The apparatus of any of Aspects 1 to 13, wherein the one ormore processors are further configured to: receive an input at an areaselection user interface (UI) area, wherein the input is one of a touchinput, a hover input, and a gesture input; and select a selection areaof the display image in response to receiving the input at the areaselection UI area, wherein the area selection UI area is distinct fromthe selection area of the display image.

Aspect 15: The apparatus of Aspect 14, wherein the selection area of thedisplay image includes at least at least a portion of the indicatorarea.

Aspect 16: The apparatus of any of Aspects 14 or 15, wherein the one ormore processors are further configured to: set an image captureparameter associated with the camera to a particular setting that isdetermined based on the selection area of the display image in responseto selection of the selection area, wherein the image capture parameterincludes at least one of a focus and an exposure parameter.

Aspect 17: The apparatus of any of Aspects 14 to 16, wherein the one ormore processors are further configured to: setting an image processingproperty to a particular setting that is determined based on theselection area of the display image in response to selection of theselection area, wherein the image processing property is one of abrightness, a contrast, a color saturation, a tone, a white balance, ablack balance, a black level compensation, a level, a gamma correction,a gain, a color correction, a noise filter, a sharpness, a blurring, anda red eye correction.

Aspect 18: The apparatus of any of Aspects 14 to 17, wherein the one ormore processors are further configured to: display an area selection UIelement at the area selection UI area.

Aspect 19: The apparatus of Aspect 18, wherein the one or moreprocessors are further configured to: generate the area selection UIelement to depict a copy of at least a portion of the selection area.

Aspect 20: The apparatus of any of Aspects 1 to 19, wherein the one ormore processors are further configured to: receive a second imagecaptured by the camera after capture of the image; determine that thesecond image is occluded by an occlusion; and display the display imageon the display in response to determining that the second image isoccluded by the occlusion.

Aspect 21: The apparatus of Aspect 20, wherein the one or moreprocessors are further configured to: receive an input, wherein theocclusion is associated with receipt of the input, wherein the input isone of a touch input, a hover input, and a gesture input; select aselection area of the display image based on the input; determine asetting based on the selection area of the display image; apply thesetting; and receive a third image captured by the camera while thesetting is applied and after capture of the second image.

Aspect 22: The apparatus of any of Aspects 1 to 21, wherein the one ormore processors are further configured to: receive a second imagecaptured by a second camera based on secondary incident light receivedby the second camera, wherein the second camera is positioned relativeto the display to receive the secondary incident light that passesthrough a second portion of the display before reaching the secondcamera.

Aspect 23: The apparatus of Aspect 22, wherein the one or moreprocessors are further configured to: determine that the second image isoccluded by an occlusion, wherein the display image is displayed inresponse to determining that the second image is occluded by theocclusion.

Aspect 24: The apparatus of any of Aspects 22 or 23, wherein the one ormore processors are further configured to: generate the display imagebased on the image and the second image.

Aspect 25: The apparatus of any of Aspects 1 to 24, wherein the one ormore processors are further configured to: determine depth informationcorresponding to one or more objects depicted in the image and in thesecond image by processing the image and the second image.

Aspect 26: A method of processing, the method comprising: receiving animage captured by a camera based on incident light received by thecamera, wherein the camera is positioned relative to a display toreceive the incident light that passes through a portion area of thedisplay before reaching the camera; displaying, on the display, adisplay image that is based on the image; and displaying an indicatoroverlaid over the display image at an indicator area of the displaywhile the display is displaying the display image, wherein the indicatorarea includes at least a subset of the portion of the display.

Aspect 27: The method of Aspect 26, wherein the method is performed by amobile device comprising the camera and the display.

Aspect 28: The method of any of Aspects 26 to 27, wherein at least oneof the indicator and the indicator area of the display are at least aslarge as an area of a lens of the camera.

Aspect 29: The method of any of Aspects 26 to 28, further comprising:receiving an input, wherein the indicator is displayed overlaid over theindicator area in response to receipt of the input.

Aspect 30: The method of Aspect 29, wherein the input includes at leastone of a touch of a touch-sensitive touchscreen surface of the display,a predefined gesture across the touch-sensitive touchscreen surface ofthe display, a hover over the display, a predefined gesture over thedisplay, a voice input reciting a predefined phrase recorded by amicrophone.

Aspect 31: The method of any of Aspects 26 to 30, further comprising:generating the display image by modifying a size of the image using atleast one of scaling and cropping, wherein modifying the size of theimage includes modifying the size of the image such that the displayimage is displayed on the display other than on the indicator area.

Aspect 32: The method of any of Aspects 26 to 31, wherein the indicatorincludes a shape around at least part of the portion of the display.

Aspect 33: The method of any of Aspects 26 to 32, wherein displaying theindicator overlaid over the display image at the indicator area whilethe display is displaying the display image includes modifying thedisplay image to merge the indicator with the display image.

Aspect 34: The method of any of Aspects 26 to 33, wherein the display isa touchscreen.

Aspect 35: The method of any of Aspects 26 to 34, further comprising:displaying an image capture user interface (UI) element over an imagecapture UI area of the display image; receiving an input at the imagecapture UI area, wherein the input is one of a touch input, a hoverinput, and a gesture input; receiving a second image captured by thecamera in response to receiving the input at the image capture UI area;and storing the second image to a non-transitory computer-readablestorage medium.

Aspect 36: The method of any of Aspects 26 to 35, further comprising:receiving an input at an area selection user interface (UI) area,wherein the input is one of a touch input, a hover input, and a gestureinput; and selecting a selection area of the display image in responseto receiving the input at the area selection UI area, wherein the areaselection UI area is distinct from the selection area of the displayimage.

Aspect 37: The method of Aspect 36, wherein the selection area of thedisplay image includes at least a portion of the indicator area.

Aspect 38: The method of Aspect 37, further comprising: setting an imagecapture parameter to a particular setting that is determined based onthe selection area of the display image in response to selection of theselection area, wherein the image capture parameter includes at leastone of a focus and an exposure parameter.

Aspect 39: The method of any of Aspects 37 or 38, further comprising:setting an image processing property to a particular setting that isdetermined based on the selection area of the display image in responseto selection of the selection area, wherein the image processingproperty is one of a white balance, a black level compensation, a gammacorrection, a gain, a color correction, a color saturation, a noisefilter, a contrast control, a brightness control, a tone mapping, asharpness, a blurring, and a red eye correction.

Aspect 40: The method of any of Aspects 26 to 39, further comprising:receiving a second image captured by the camera after capture of theimage; determining that the second image is occluded by an occlusion;and displaying the display image on the display in response todetermining that the second image is occluded by the occlusion.

Aspect 41: The method of Aspect 40, further comprising: receiving aninput, wherein the occlusion is associated with receipt of the input,wherein the input is one of a touch input, a hover input, and a gestureinput; selecting a selection area of the display image based on theinput; determining a setting based on the selection area of the displayimage; applying the setting; and receiving a third image captured by thecamera while the setting is applied and after capture of the secondimage.

Aspect 42: The method of any of Aspects 26 to 41, further comprising:receiving a second image captured by a second camera based on secondaryincident light received by the second camera, wherein the second camerais positioned relative to the display to receive the secondary incidentlight that passes through a second portion of the display beforereaching the second camera.

Aspect 43: The method of any of Aspects 26 to 42, further comprising:determining that the second image is occluded by an occlusion, whereinthe display image is displayed in response to determining that thesecond image is occluded by the occlusion.

Aspect 44: The method of any of Aspects 26 to 43, further comprising:determining depth information corresponding to one or more objectsdepicted in the image and in the second image by processing the imageand the second image.

Aspect 45: A non-transitory computer-readable medium having storedthereon instructions that, when executed by one or more processors,cause the one or more processors to perform operations according to anyof aspects 1 to 44.

Aspect 46: An apparatus comprising one or more means for performingoperations according to any of aspects 1 to 44.

What is claimed is:
 1. An apparatus for imaging, the apparatuscomprising: at least one memory; and at least one processor coupled tothe at least one memory and configured to: receive an image of a scenecaptured by a camera based on incident light received from the scene bythe camera, wherein the camera is positioned relative to a display toreceive the incident light from the scene that passes through a portionof the display before reaching the camera; display, on the display, adisplay image that is based on the image; and receive, using a userinterface (UI), a UI input configured to select a selection area of thedisplay image, wherein at least a portion of the selection area of thedisplay image is displayed on the portion of the display while thedisplay image is displayed on the display.
 2. The apparatus of claim 1,wherein the apparatus is at least one of a mobile device, a wirelesscommunication device, or a camera device.
 3. The apparatus of claim 1,wherein the apparatus includes at least one of the camera or thedisplay.
 4. The apparatus of claim 1, wherein the display image is theimage.
 5. The apparatus of claim 1, wherein the display image is aprocessed variant of the image.
 6. The apparatus of claim 1, wherein thedisplay is a touchscreen display, wherein the UI includes atouch-sensitive layer of the touchscreen display, and wherein the UIinput includes a touch input detected by the touch-sensitive layer ofthe touchscreen display.
 7. The apparatus of claim 1, the at least oneprocessor configured to: receive a second image captured by the cameraafter capture of the image; and determine that the second image is atleast partially occluded by an occlusion, wherein the display image isdisplayed on the display based on the second image being at leastpartially occluded by the occlusion.
 8. The apparatus of claim 7,wherein the occlusion is associated with receipt of the UI input,wherein the UI input includes at least one of a touch input, a hoverinput, or a gesture input.
 9. The apparatus of claim 7, the at least oneprocessor configured to: determine a setting based on the selection areaof the display image; apply the setting; and receive a third imagecaptured by the camera while the setting is applied and after capture ofthe second image.
 10. The apparatus of claim 1, the at least oneprocessor configured to: determine a setting based on the selection areaof the display image; apply the setting; and receive a second imagecaptured by the camera while the setting is applied and after capture ofthe image.
 11. The apparatus of claim 10, wherein the setting is appliedto an image capture parameter, wherein the image capture parameterincludes at least one of a focus parameter or an exposure parameter. 12.The apparatus of claim 1, the at least one processor configured to:determine a setting based on the selection area of the display image;receive a second image captured by the camera; and apply the setting tothe second image.
 13. The apparatus of claim 12, wherein the setting isapplied to an image processing parameter, wherein the image processingparameter includes at least one of a brightness, a contrast, a colorsaturation, a tone, a white balance, a black balance, a black levelcompensation, a level, a gamma correction, a gain, a color correction, anoise filter, a sharpness, a blurring, or a red eye correction.
 14. Theapparatus of claim 1, wherein the UI includes a button, and wherein theUI input includes a press of the button.
 15. The apparatus of claim 1,wherein the UI input is received at an area selection UI area of thedisplay that is distinct from the selection area of the display image,wherein the UI input includes at least one of a touch input, a hoverinput, or a gesture input.
 16. The apparatus of claim 15, wherein thedisplay is a touchscreen, wherein the area selection UI area includes avirtual touchscreen button, wherein the UI input includes a button pressof the virtual touchscreen button of the area selection UI area.
 17. Theapparatus of claim 15, wherein the area selection UI area is configuredto control a cursor, wherein the UI input includes use of the areaselection UI area to manipulate the cursor to indicate the selectionarea.
 18. The apparatus of claim 15, wherein the at least one processoris configured to: display an area selection UI element at the areaselection UI area.
 19. The apparatus of claim 18, wherein the at leastone processor is configured to: generate the area selection UI elementto depict a representation of at least a portion of the selection areaof the display image.
 20. The apparatus of claim 1, the at least oneprocessor configured to: display an indicator overlaid over the displayimage at an indicator area of the display while the display isdisplaying the display image, wherein the indicator area overlaps withthe portion of the display through which the incident light passes,wherein the indicator is indicative of a position of the camera, whereinthe selection area of the display image includes at least a portion ofthe indicator area.
 21. The apparatus of claim 1, the at least oneprocessor configured to: determine a depth of an object in the scenebased on the selection area of the display image depicting at least aportion of the object; and determine a setting based on the depth of theobject, the setting configured to be applied to a second image to becaptured using the camera.
 22. The apparatus of claim 1, the at leastone processor configured to: determine an attribute of a depiction of anobject in the display image, the object being part of the scene, thedepiction of the object overlapping with the selection area of thedisplay image, the attribute including at least one of a brightness, alevel of contrast, a sharpness, a level of blur, a color, a hue, asaturation level, a tone, or a level of noise; and determine a settingbased on the attribute of the depiction of the object, the settingconfigured to be applied to a second image to be captured using thecamera.
 23. A method of imaging, the method comprising: receiving animage of a scene captured by a camera based on incident light receivedfrom the scene by the camera, wherein the camera is positioned relativeto a display to receive the incident light from the scene that passesthrough a portion of the display before reaching the camera; displaying,on the display, a display image that is based on the image; andreceiving, using a user interface (UI), a UI input configured to selecta selection area of the display image, wherein the selection area of thedisplay image is displayed on the portion of the display while thedisplay image is displayed on the display.
 24. The method of claim 23,wherein the display is a touchscreen display, wherein the UI includes atouch-sensitive layer of the touchscreen display, and wherein the UIinput includes a touch input detected by the touch-sensitive layer ofthe touchscreen display.
 25. The method of claim 23, further comprising:receiving a second image captured by the camera after capture of theimage; and determining that the second image is at least partiallyoccluded by an occlusion, wherein the display image is displayed on thedisplay based on the second image being at least partially occluded bythe occlusion.
 26. The method of claim 23, further comprising:determining a setting based on the selection area of the display image;applying the setting; and receiving a second image captured by thecamera while the setting is applied and after capture of the image. 27.The method of claim 23, further comprising: determining a setting basedon the selection area of the display image; receiving a second imagecaptured by the camera; and applying the setting to the second image.28. The method of claim 23, wherein the UI includes a button, andwherein the UI input includes a press of the button.
 29. The method ofclaim 23, wherein the UI input is received at an area selection UI areaof the display that is distinct from the selection area of the displayimage, wherein the UI input includes at least one of a touch input, ahover input, or a gesture input.
 30. A non-transitory computer-readablemedium having stored thereon instructions that, when executed by one ormore processors, cause the one or more processors to: receive an imageof a scene captured by a camera based on incident light received fromthe scene by the camera, wherein the camera is positioned relative to adisplay to receive the incident light from the scene that passes througha portion of the display before reaching the camera; display, on thedisplay, a display image that is based on the image; and receive, usinga user interface (UI), a UI input configured to select a selection areaof the display image, wherein the selection area of the display image isdisplayed on the portion of the display while the display image isdisplayed on the display.